DOCUMENT RESUME 



ED 214 995 

AUTHOR 
TITLE 

INSTITUTION 

SPONS AGENCY 
PUB DATE 
CONTRACT 
NOTE 



EDRS PRICE 
DESCRIPTORS 



TM 820 252 

Spandel, Vicki 

Classroom Applications of Writing Assessment: A 
Teacher's Handbook. 

Northwest Regional Education Lab., Portland, Oreg. 
Clearinghouse for Applied Performance Testing. 
National Inst, of Education (ED), Washington, D.C. 
May 81- 
400-80-0105 • 

97p.; For related documents see ED 196 038, ED 195 
576, and ED 213 035. 

MF01/PC04 Plus Postage. 

Elementary Secondary Education; "Evaluation Methods; 
♦Holistic Evaluation; *Scoring; Teachers; "Training 
Methods; Writing (Composition); "Writing 
Evaluation f . 

IDENTIFIERS "Analytic Scoring; "Primary Trait Scoring 

ABSTRACT 

This handbook on Classroom Applications of Writing 
Assessment represents the fourth and final entry in a series of 
interrelated publications on the assessment of writing proficiency 
prepared over the past year by the Clearinghouse for Applied 
Performance Testing (CAPT). This handbook was written-40 meet the 
needs of the classroom teacher who is teaching writing) and who 
wishes to incorporate performance assessment strategies into that 
instruction. Specifically, the handbook provides classroom teachers 
with (1) background information on the basic principles of three 
strategies for evaluating student writing samples: holistic, 
analytical and primary trait scoring; (2) extensive experience in the 
application of these scoring strategies to hypothetical samples of 
student writing; and (3) a series of very practical and immediately 
useful ideas for using these scoring strategies in day-to-day writina 
instruction. (Author/GK) * 



********************************************************* # ^*****^^^ 

* Reproductions supplied by EDRS are the best that can be made * 

* from the original document. * 
****************************** ***************************************** 



"PERMISSIONTO ftEPRODUCC tu^ 



if* OWAmMHVT OMOUCATlON, 

. NATIONAL WSmUTCOFWOCAnOli 

EDUCATIONAL RESOURCES' INFORMATION 
CENTER* (EPIC) 

r*c*v«jj from p« r5on oroaniMtkii* 

'•PfoAictiort quality * * * 
• ^omt.ofv^ WOP K> 1 on $ ,tat«f»nth lt docu- 



Classroom Applications 
of Writing Assessment 

A TEACHER'S HANDBOOK 



Northwest Regional Educational Laboratory 



Classroom Applications 
Of Writing Assessment: 



A Teacher's Handbook 



Vicki Spandel 



CAPT 



Clearinghouse for Applied Performance Testing 

NORTHWEST REGIONAL EDUCATIONAL LABORATORY 
300 S.W. Sixth Avenue 
Portland, Oregon 97204 



3 



May 1981 



This work is published by the Clearinghouse for Applied Performance Testing (C APT) of the 
Northwest Regie ial Educational Laboratory, a private nonprofit corporation. The work 
contained herein has been developed under contract 400-80-0106 with the National Institute 
. of Education (NIE), U.S. Education Department, the opinions expressed in this publication 
do not necessarily reflect the position of the National Institute of Education, and no official 
endorsement by the Institute should be inferred. 

CAPT Director: Beverly Anderson 

NIE Project Monitor: The! Kocher 

Advisory Board: 
Richard Brickley 
Dale Carlson 
Christine McGuire 



PREFACE 



This handbook on Classroom Applications of Writing Assessment represents the fourth 
and final entry in a series of interrelated publications on the assessment of writing proficiency 
prepared over thepast year by the Clearinghouse for Applied Performance Testing (CAPT). ' 
This handbook was written to meet the needs of the classroom teacher who is teaching writing, 
and who wishes to incorporate performance assessment strategies into that instruction. 

The first in CAFFs writing assessment series is the book Direct Measures of Writing 
Skill: Issues and Applications, by Vicki Spandel and Richard Stiggins, published by the Nor- 
thwest Regional Educational Laboratory in January 1980. That book provides a comprehensive 
analysis of large-scale writing assessment procedures used around the country. Highlights in- 
clude a comparison of five writing sample scoring procedures, and results of a national survey 
of statewide writing assessment programs. This book is aimed at the educator who wants 
background information on the most successful ways developed to date for assessing writing. 

The second publication in the series, Writing Assessment in the 1980s, presents the pro- 
ceedings of a CAPT 1980 training seminar on writing assessment. The proceedings include 
three writing assessment specialists' predictions for future directions in writing assessment, 
along with abstracts of training experiences offered to seminar participants. 

The third publication, A Consumer's Guide to Writing Assessment by Nancy Bridgeford 
and Richard Stiggins, contains information of value to those interested in measuring writing 
proficiency. First, the guide presents a detailed comparison of two measurement approaches: 
the direct method (using writing samples) and the indirect method (using objective tests of 
language usage proficiency). Advantages and drawbacks of each are discussed to aid potential 
assessors in deciding which approach best meets their needs, Second, the guide lists well over 
200 educators nationwide capable of providing technical assistance in developing writing tests. 
Background, experience and contact information are provided. Third, the guide lists over 50 
published tests that can be used as indirect measures of writing proficiency. 

With this series of four publications, CAPT hopes to aid educators at various levels in 
dealing with both large-scale and small-scale writing assessment. We hope you will find this 
handbook helpful, and that its use will encourage you to explore other CAPT publications. 
Clearinghouse staff welcome your comments and inquiries. 

Richard J. Stiggins % 
CAPT Coordinator 

Beverly L. Anderson 
Project Director 
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PRETEST 

To see how much you already know about applying writing assessment scoring techni- 
■ ques in the classroom, we invite you to take the pretest. You will have a chance to repeat the 
test after completing the handbook materials so that you can compare your response. 
Answers are provided in Appendix G. 

If you are using these materials in a group workshop, print two copies of the test (for pre- 
test and posttest) for each participant. 

PRETEST 
Test of Writing Assessment Knowledge 

/NAME: _ SCORE: BEFORE TRAINING 

1. List the major steps in the writing assessment process. 

2. List three different approaches to scoring samples of student writing. 

3. Name the scoring approach that relies on an overall impression of the quality of a given 
piece of writing. ' 

4. Name the scoring approach based on the premise that all writing is done to com- 
municate with a specific audience. 

5. Which scoring approach takes the most time to use? 

6. Which scoring approach takes the least time to use? 

7. Which scoring approaches are based on prespecified performance criteria? 

8. If you wanted to evaluate specifically how well students follow proper format in writing 
a business letter, which scoring approach would you use? 

9. Which scoring approach provides thje most information about specific strengths and 
weaknesses in student writing? 

10. What is a "range finder" paper? 

11. Which of the scoring approaches is probably most subject to a "halo effect"? 

12. Probably the best way to use writing assessment in the classroom is to show students 
how you are going to score their papers. 

True 

False 



DO NOT TURN TO THE ANSWERS (APPENDIX G) UNTIL YOU HAVE ALSO COM- 
PLETED THE POSTTEST. 
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CHAPTER I: Using Writing Assessment 

in the Classroom 



PURPOSES OF THE HANDBOOK 

The climate of today's classroom reflects a growing demand for more emphasis on basic 
skills. Teacher^of writing feel the pressures of this demand strongly, and seek every means avail- 
able to make their instruction more effective. As writing assessment gains prominence, many 
teachers are beginning to use alternative methods of writing assessment both as part of instruc- 
tion and as a means of helping them make decisions based on student performance. 

The ways .in which writing assessment can be used in the classroom are many and varied. 
Some teachers have found that teaching students to evaluate one another's writing makes them 
better critics of their own work and, therefore, better writers. Others have found that the sys- 
tematic evaluation of student writing - using the kinds of scoring approaches discussed later in 
this handbook - can be very useful in diagnosin^individual students' strengths and weaknesses. 
And still others use writing tests to select students for placement into advanced or remedial writ- 
ing courses. This handbook will illustrate additional ways writing assessment can be used to ad- 
vantage in the teaching of basic writing skills. 

Specifically, the handbook provides classroom teachers with (1) background information 
on the basic principles of three strategies for evaluating student writing samples: holistic, analyt- 
ical and primary trait scoring; (2) extensive experience in the application of t)£se scoring strate- 
gies to hypothetical samples of student writing; and, (3) a series of very practical and immediately 
useful ideas for using these scoring strategies in day-to-day writing instruction. 

CAPT recognizes that many teachers already employ the scoring approaches described in 
this handbook as part of their regular review of student papers. You may find much here that is 
familiar in theme even if it is presented under a formal label (e.g., primary trait scoring) that you 
would not customarily apply to your own work. The purpose of any training in scoring techniques 
for writing assessment is not, after all, to provide teachers with expertise in evaluating papers; 
presumably, teachers of English have already acquired such expertise through study and experi- 
ence. The purpose, rather, is to help teachers gain consistency in the way they approach that * 
evaluation. In the classroom, such consistency clarifies expectations for students, and makes it 
easier for both student and teacher to set goals and to know when those goals hav* been reached. 



FOCUS OF THE HANDBOOK 

There are two viable approaches to the measurement of writing skill in the classroom. 
First, the teacher can use actual samples of student writing to evaluate writing skill directly. 
Second, the teacher can use one of the many published tests of language usage proficiency to 
measure writing skill indirectly. The many differences and similarities between these two ap- 
proaches are addressed in detail elsewhere. This handbook focuses only on the first approach: 
measuring proficiency via the evaluation of actual samples of student writing. 
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There are three specific reasons for this. First, although direct measures are very easy 
for teachers to use, and most are experienced in administering such measures, teachers are 
Really not trained to develop their own objective tests of language usage proficiency. 
Therefore, those wanting to use objective measures must generally rely on published tests 
which are often too expensive to make daily use practical. Items for direct measure of writing 
skill, on the other hand, can be readily developed by teachers at no cost whatever. 

Second, teacher-made writing exercises can be designed to focus on a particular aspect of 
writing. Published objective tests, of necessity general in scope, do not meet a teacher's day-to- 
day instructional needs nearly so well. 

Third, direct measures require students to write, whereas indirect measures frequently 
require only fill-in or multiple-choice responses. Direct measures, therefore - as used in the 
classroom - are not so much tests per se as they are opportunities for students to improve 
writing skills through practice. 

Audience 

. This handbook is directed to those teachers who have responsibility for teaching writing 
skills. It is expected that handbook users will have formal training arid experience in English 
education and some experience with the assessment of writing skill. However, a background or 
experience in educational testing is not assumed. 



STEPS IN THE ASSESSMENT PROCESS 

Though the handbook deals almost exclusively with the scoring of writing samples, the 
user should be aware that scoring is only one step (albeit a very important step) in, the writing 
assessment process. To put scoring into perspective, we shall briefly review the three 
preceding steps. 

Step 1: Clarifying the reason for measuring writing skill. No test should ever be ad- 
ministered until the user has clearly in mind how results will be used. For example, writing as- 
sessment can be used in the classroom to aid student development; the process of criticizing 
one's own or another's writing can be instructive. Writing assessment can also help teachers 
make decisions. For example, the teacher can use assessment to diagnose strengths and weak- 
nesses for placement purposes, to certify minimum acceptable levels of proficiency, or to 
evaluate a particular program of instruction. This clarification is essential because the reason 
for assessment has implications for the way a test item is structured, administered and scored. 

Step 2: Planning the writing exercise. The writing exercise (or item) is key to the suc- 
cess of any assessment. The item should provide clear instructions that identify the purpose for 
writing, the audience to be addressed and the form of writing to be produced (e.g., business let- 
ter, essay). The stimulus may be given orally or in written form (or both), and may - especially 
if it adds interest - include a picture. A good item is clear, understandable, interesting and 
bias-free, thus giving the student every opportunity to respond to his or her fullest capability. 

Step 3: Administering the writing assessment It is always best to administer a writing 
test under conditions that approximate real world circumstances. To the extent that this condi- * 
tion is met, the teacher can feerconfident that the proficiency demonstrated by the students is 
reflective of what would occur in *eal life. Most real-world writing includes three factors; (1) 
choice of the content (within limits); (2) reasonable time limits free of undue pressure; and (3) 
time to reread, rethink and revise the message. Consideration should be given to including 
these conditions to the extent possible. If we impose severe time limits, require rigid content 
and fail to allow opportunity for revision, we may succeed in standardizing test circumstances^ 
while missing the mark on something far more critical: that is, establishing testing condi^ns 
that sufficiently approximate real Fife to make results meaningful. "/ 

Once one has determined the purpose for testing, designed and administepedthe test 
item, one is ready to score results according to the approach that best suits that testing nurpose 
and design. 

n 
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SCORING APPROACHES 



For all writing sample scoring methods, raters must be trained to score papers according 
to specific procedures. The purpose of training is to minimize (at least temporarily) the effects 
of individual biases (e.g., overemphasis on mechanics or content) by helping raters internalize 
scoring standards. , 

The exercises provided in this handbook offer practical experience with holistic, 
analytical and primary trait scoring. Though these three scoringallernatives were developed 
for use in major large-scale writing assessments by the National Assessment of -Educational 
Progress and the Educational Testing Service, each has significant potential for use in small- 
scale in-class writing assessments. The three approaches are described briefly here, arid in 
more detail in subsequent chapters, - - ' . 

Holistic scoring. Imholistic scoring, the rater (teacher) reviews a paper for an overall or 
"whole", impression. Specificvfactors such as grammar, usage, style, tone and vocabulary un- 
doubtedly affect the rater's response, but none of these factors is isolated and scored separate- 
ly- \ 

Raters use "model" papers, called range finders, as guides. These model papers are taken 
from th- sample of papers to be scored. The process by which they are selected is explained in 
detail in Chapter II: Holistic Scoring Practice, and also in Appendix B. 

Holistic scoring is rapid and efficient. Depending on the length of student responses, ex- 
perienced raters can often go through 30 to 40 papers per hour. 

Because no explicit scoring criteria are set (remember, model papers provide the guide- 
lines for scoring), holistic scoring does not permit the reporting of specifics on student perfor- 
mance. That Is, one cannot say that a certain percentage of students did well on mechanics or 
poorly on organization.. After reading a number of papers, however, teachers typically have a 
supremely clear notion of what factors influenced them to assign particular scores. 

Analytical scoring! Analytical scoring, in direct contrast to holistic scoring, involves 
isolating one or more characteristics of writing and scoring them individually. Analytical scor- 
ing is most appropriate if one wants to measure (and report) students' ability to deal with 
specific aspects of writing: punctuation, organization, syntax, usage, creativity, sense of au- 
dience, and so on. Criteria for assigning scores must be explicit and complete, and must be well 
understood by all raters. Any traits can be selected for scoring, provided they have universal 
application. That is, they must be traits important to any piece of writing in any context - from 
;a movie review to a valedictory speech. 

Analytical scoring provides data on specificaspects of student writing performance. But 
does it redly reveal whether, in general, students write well? The answer depends on (1) 
whether enough traits are analyzed to provide a comprehensive picture, and (2) whether those 
traits analyzed are significant: that is, whether they actually contribute to good writing. 

In Chapter III, you will have a chance to score student papers on three significant traits: 
ideas, wording and organization. This practice should be sufficient to acquaint you with the 
basics of analytical scoring. To obtain a more comprehensive analysis of a paper's quality, you 
might, however, wish to score additional traits, such as style. 

In holistic scoring only a single-score is assigned. That score indicates a paper's general 
quality, all traits considered in balance. In analytical scoring, each trait is scored separately; the 
overall effectiveness of a paper is presumably reflected in the sum of these scores. 

Primary trait scoring. Primary trait scoring is similar to analytical scoring in that it fo- 
cuses on a specific characteristic (or characteristics) of a given piece of writing. However, while 
analytical scoring attempts to isolate those characteristics important to any piece of writing in 
any situation, primary trait analysis is Situation-specific. The most important - or primary - 
trails) irr a letter to the editpr will not likely be the same as that (those) in a set of directions for 
assembling a bicycle. 
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To look at it another way, the primary trait system is based on the premise that all writ- 
ing is done in terms of an audience, a*id that successful writing will have the desired effect upon 
that audience. For example, a good mystery story will excite and entertain the reader; a good 
letter of application will get a job interview. In a scoring situation, of course, papers must be 
judged on the likelihood of their producing the desired response. 

Because they are situation-specific, primary fraits differ from item to item, depending on 
the nature of the assignment. Suppose a student were asked to give directionsfor driving from 
his/her home to school. The primary trait might then be sequential organization, for any clear, 
unambiguous set of directions would Jfcssarily be well organized with details presented in 
proper order, v 

Like analytical scoring, primary trait scoring can allow the reporting of student perform- 
ance with respect to specific characteristics: e.g., organization, awareness of audience. For this 
reason, primary trait and analytical scoring are often favored over holistic scoring in contexts 
where precise information is needed. But this advantage must be carefully weighed against the 
time and effort required to scorr papers analytically or using primary trait analysis. The more 
individual traits scored, the more time required for scoring. 



CONCLUSION 

Many teachers will find all three scoring approaches useful in classroom application for 
various purposes. Holistic scoring practice provides an excellent base for open-ended discussion 
about what makes a particular piece of writing good -or not so good. Through such practice 
students learn to view their own and others' writing with a new critical eye. Analytical scoring 
practice refines that critical skill by offering students specific guidelines for making writing ef- 
fective. And finally, primary trait scoring practice centers on the importance of writing to an 
audience. 



USING THE HANDBOOK 

The handbook can be used in either of two wa>s: (1) as a programmed individualized in- 
structional program or (2) as the basis for a group workshop on writing assessment in the class- 
room. 

If you are using it as an individual study guide, simply proceed through the background 
material and exercises at your own pace and you will gain the needed skills to use writing as- 
sessment successfully in your writing instruction. Please note, however, that it may be 
helpful for you to find a colleague with whom to share reactions, compare scores and 
discuss the ideas presented. 

If you would like to use these exercises in a group presentation, or workshop, please ton 
to Appendix F for more specific instructions on doing so. * 



NOTE: We are eager to make this handbook as timely avd useful a resource 
as possible. Therefore, if you have aMitioncd suggestions on how any of 
these scoring amrroaches might be used successfully %n class, won't you let ics 
hear from your We shall endeavor to incorporate any comments, sugges- 
tions or anecdotes into future updates of this handbook. 




CHAPTER II: Holistic Scoring Practice 



This chapter contains a sample test item and hypothetical student responses to that item. 
You wall be asked to score the hypothetical responses holistically, then to compare your scores 
with those suggested by the author.* Given practice, and an opportunity to compare your 
reasoning with the rationales provided by the author, it is anticipated that you will achieve a 
high level of agreement with the scores provided by the . author. Occasional disagreement 
should not be cause for concern; raters are individuals and no amount of "training" will erase all 
differences in their ways of approaching scoring. Consistent disagreement, however, should en- 
courage you to carefully review your own personal guidelines for assigning scores; some sug- 
gestions: for doing so will be offered. 

Keep in mind that the primary purpose of this handbook is not to train you as a rater. 
However, only by understanding fully how to score papers, and by practicing various scoring 
approaches, can you feel confident in using scoring techniques as a part of instruction. 



HOW TO SCORE 

Item 1, (page 8 ) was written to be scored holistically. Holistic scoring involves reading a 
paper quickly for an overall or "whole" impression. While specific traits such as organization, 
syntax, originality and mechanics undoubtedly influence (and should influence) a rater's judg- 
ment, none of these traits is individually, directly addressed. What matters is how all traits 
work in harmony to produce the final piece. In a given paper, for instance, spelling may be very 
poor, punctuation fair, word choice exceptional, organization above average, originality strik- 
ing. So perhaps the rater will determine that strengths outweigh weaknesses, and will assign 
this paper a fairly high score. In general, a rater makes a judgment in much the same way that 
he or she decides whether a novel or an essay in Time magazine is superior, mediocre, or slip- 
shod. Holistic scoring relies on impressions, not on rigorous line-by-line scrutiny. 

Holistic raters do not make marks on the papers they score. There are two primary rea- 
sons for this. Firs w in an actual scoring situation, a rater cannot afford to take time to mark pa- 
pers in this fashion. Each paper is scored within a minute or two. Second, marking obvious er- 
rors often directs too much attention to one aspect of a paper (such as punctuation), while 
allowing a rater to virtually ignore other less tangible, but equally important features (such as 
style). All aspects of a paper must be considered in balance. 

For purposes of this exercise, a 4-point rating scale will be used, with 4 being the upper 
limit of the scale. Essentially, you will be using your own judgment, based on your experience 
as a language arts educator, in assigning scores. However, to help you, four ."model" papers 
called ranqe finders are provided. A range finder is a paper selected on the basis of its repre- 
sentativeness for use as a scoring guide. Range finders are generally selected from within the 



'Materials in this handbook have been field tested with language arts teachers, and suggested scores are based in 
part on the results of that field testing. 



sample to be scored. With a 4-point scale, there is one range finder for each of the score levels: 
4, 3, 2 and 1. Range finder papers are considered so typical of papers at a given score level that 
virtually all trained readers would agree on the score each range finder should receive. In other 
words, you as a rater can feel confident that if a paper demonstrates about the same skill as a 
level 3 range finder, that paper should receive a score of 3. 

A range finder paper represents about the midpoint of a scoring range. In other words, a 
level 4 range finder is not considered the best single paper from among a given group of 
papers. Rather, it is superior to some 4s, but not as good as ethers. If one pictures the rating 
scale as a continuum, the level 4 range finder occurs about midway between the cutoff points 
for 3 and 4 . F 

When You're Ready to Score. 

To begin, read Item #1 through carefully. Then study each range finder, along with the 
scoring rationales provided in the subsequent discussion. When you are ready, turn to page 11 
to score papers A through E,. 

HOLISTIC SCORING: Sample Test Item 

Following is a sample test item that might be given to a group of eighth graders* to mea- 
sure general writing proficiency. You can assume that students would have as much time as they 
needed to respond, and would be allowed to make collections on their papers if they wished. Also 
assume that instructions would be given orally as well as in writing.** 



Item 1 

Toby loved to hike in the woods. One day when he was hiking, he found a 
bird with an injured wing. Toby took the bird home, set his wing, and took 
care of him. He named the bird Mike. Soon Mike got better, and he could 
have flown away, but Toby kept him in a small cage. Toby knew the bird was 
ready to be free, but he liked Mike so much he decided to keep him as a pet. 

Do you think Toby made the right decision? Write your answer on this page. 
State clearly whether you think it was a good idea for Toby to keep Mike. 
Then explain your reasons as completely as you can. 



* Items in this handbook are considered adaptable for use with any grade level. 

••Giving instructions both orally and in writing helps ensure that an item truly measures writing - and not reading 
skills. 
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Range Finders for Item 1 



Level 4 Range Finder:* Toby has definitely not made the right decision. Unless we all protect 
our wildlife, generations to come will have no wild birds and animals to enjoy. Toby meant well 
m fixing the bird's broken wing. But he went too far in trying to turn a wild creature into a 
domestic pet. No one can care for a wild bird or animal better than Nature Itself. What if every- 
one did what Toby has done? Soon we would have no wild creatures left. It is only natural to 
want a pet. But it is selfish to keep a wild bird confined just for your own pleasure. A pet may 
bnng one person pleasure, but a wild c-eature can bring pleasure to everyone who sees and en- 
joys it. I believe Toby should set Mike free. 

Level 3 Range Finder: In my judgment, Toby has made a poor decision. Wild birds are not pets 
and are not meant to be pets. It is easy to see how Toby would get to like the bird after caring for 
it and helping it to get well. That does not give him the right to keep this bird. Mike was a wild 
thing with the right to remain free. Even things as small as birds have rights and feelings. Toby 
did a good thing to help the bird, but once Mike was well Toby should let him go. 

Level 2 Range Finder: Here is a hard choice. For if a person love his pet it is hard to let the pet 
go. Toby loves Mikftand he is the one who found him. So maybe if the bird is well and then he 
made him well it is all right to keep him for a while. But he should build a bigger cage for Mike 
so he will not get cramped and be sure to feed him the right things. Then if Mike stays happy 
and stays there of his own free will I say sure its OK to keep him for a while (But Mike is not the 
best name for a bird.) 

Level 1 Range Finder: No. Birds do not mak good pets I think. Soon Mike could get sick and die, 
then Toby will be sorry. A dog lifes longer than birds and gives better compony so that is what 
Toby should do. 



All student papers used in this handbook are typed rather than handwritten to enable users to score quickly Readers 
are asked to keep in mind that the purpose of these materials is to familiarize users with scoring procedures, not to train 
them as professional raters; therefore, practice in coping with the variable of handwriting, over and above learning to 
apply scoring procedures, was deemed unnecessary. For a discussion of how handwriting affects scoring see Appendix 
A, Question 5. For examples of handwritten student responses, see Appendix C. 
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Discussion of Range Finders 

NOTE: You may find it helpful at this point to remove the four range 
finders from your handbook so that you can keep them in frorU of you for 
handy reference. 

To better understand what's behind these ratings, let's take a closer look at a few of the 
strengths and weaknesses of each range finder paper. Well begin this time at the low end of the 
scale, with the level 1 range finder. 

Level 1 Range Finder: Even a cursory reading reveals faulty spelling and punctuation. But 
there are more serious problems: weak sentence structure, faulty shift in mood (could to will), 
vague pronoun reference ("that is what Toby should do"). Moreover, the response addresses the 
issue only briefly, then switches topics; the question is whether Toby made a sound decision, not 
whether birds make good pets. Brevity per se is not a fault; in this instance, however, the re- 
sponse is not concise, but rather iacking in content. "No" does not constitute a clearcut stand on 
an issue. 

Level 2 Range Finder: The opening sentence is weak; it offers a vague reference rather than 
the strong statement of position called for. The final sentence contains minor punctuation er- 
rors. The writer waffles a bit ("maybe ; . . it is all right to keep him for a while'O, but finally con- 
cludes that Toby's choice is "OK," and says so with reasonable clarity. The reference to building 
a bigger cage hedges the issue a bit, but does show somewhat more thought than the writer of 
paper #1 has g^ven the topic. This is better than a 1 paper because the writer has made a clear 
attempt to be responsive, but the paper still lacks direction, style, and sense of control. 

Level 3 Range Finder: Paper #3 offers an adequate, though not particularly imaginative re- 
sponse. The paper starts out strong, but is weakened by redundancy and limited reasoning: 
that is, the writer cannot progress beyond the logic that Mike should be &et free because wild 
creatures should be free (a circular argument). On the positive side, the sentence structure is 
fair, aside from a troublesome shift in tense in the last sentence. Word choice is not striking, 
but it does the job. The writer displays fair control of thought and language. The style is a little 
flat, but the paper has a central theme and gets the point across clearly, if not concisely. 

Level 4 Range Finder: This writer responds directly and fully to the issue. The opening sen- 
tence clarifies a position that never wavers throughout the paragraph. Moreover, the writer 
provides sound reasoning to back up the stated position (i.e., protection of wildlife), and ex- 
pands this thinking to demonstrate some awareness of larger implications ("What if everyone 
did what Toby has done?"). The result is a clear, thoughtful, well handled piece, free of redun- 
dancy. Word choice is good, sentence structure sound. 

Summary 

The preceding discussion illustrates what might occur to a rater in assigning scores. Addi- 
tional strengths and weaknesses may have occurred to you in reading through the range finders. 
Rating papers is a complex task. Each paper is unique; moreover, each rater has his or her own 
biases (e.g., one attends closek to mechanics, another to style, and so on). The purpose of scoring 
practice is to minimize those mases so that raters can come as close as possible to agreement in 
assigning scores. 
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UNIT 1: HOLISTIC SCORING PRACTICE (PAPERS A-E) 



Unit 1 includes five papers, A through E. Score each paper holistically on a 4-point scale, 
referring often to the range finders provided for guidance. You are encouraged to avoid making 
editorial marks on the papers, and to score papers quickly, trusting your impressions. 

Following the five student papers, the author presents suggested scores for each paper and 
a rationale for each score assigned. After scoring, compare your own reasoning with that of the 
author to detennme extent of agreement. Do NOT turn to the Rationale section until you have 
scored ALL FIVE PAPERS, A through E. 

NOTE: If for any reason you think you may wish to reuse these materials, or 
share them with a colleague, we suggest you make copies of student responses 
oejore scoring. 

Sample Scoring Exercises (A-E) 

Enter your score in the blank provided. 
PAPER A SCORE: 

We should try to see things from Toby's side. I mean, it isn't every day that you get a chance to 
study nature up close. Now Toby has found this wild bird and he has a chance to study it and ob- 
serve it. Maybe it is not the best thing to happen for a bird to be kept in a cage, but how else is he 
going to learn from it. After a while, he can let the bird go. 

PAPER B SCORE: 

I am sorry to say it but Mike has made a poor decision. Wild birds do not live long in captivity, as 
a rule. They do not receive the right food or enough exercise to keep their wings strong. What 
can Mike know about fixing a broken wing? Due to his carelessness, Toby may never fly again. 
He should have taken the bird to a qualified doctor so he could be properly treated and then re- 
leased. 

PAPER C SCORE: 

Here is what I would do. Take the bird to the zoo and there let it get fixed up. So then it would be 
beter then the zoo could keep the bird 

PAPER D SCORE: 

Toby faces a dilemma. If he keeps the bird, he is depriving a wild creature of its freedom. If he 
lets the bird go, he is taking a chance that the bird will no longer be able to care for itself in the 
wild. Once tamed, a wild creature often becomes dependent. Toby should release the bird, but 
wateh him carefully for several days to be sure he can care for himself. Then he will know he has 
made the right decision. His decision to keep the bird as a pet is not a responsible one and I'm sure 
that when he reconsiders he will change his mind. 

PAPER E SCORE: 

Keeping wild birds in captivity is illegal. Therefore, Toby made the wrong decision. 



PLEASE TURN THE PAGE FOR SUGGESTED SCORES AND RATIONALES. 
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UNIT 1: SUGGESTED SCORES AND RATIONALES (PAPERS A-E) 



PAPER A Suggested Score: 2 

Rationale: With the first two sentences, this paper looked like a 3. The writer hit on a fresh idea: 
keeping the bird for observation. However, the idea is never developed. In addition, the writer 
never clearly states a position with regard to Toby's decision, though we can infer he is in favor of 
Toby's keeping the bird until the last sentence, which weakens even that implied position. Mech- 
anics and sentence structure are reasonably sound. Vocabulary is adequate, though not excellent. 

Did you agree with the author's rating? Remember that in holistic scoring, a 1-point discrepancy 
is generally considered "agreement." This is easy to understand when one realizes that a paper at 
the upper end of, say, the 2-point range is actually closer to a low 3 than to a low 2 There is 
space below for you to note your agreement/disagreement and any comments you may have. 
These notes may be of value to you later in constructing classroom exercises/discussions. 

Agree 

Disagree 

Comments: 



PAPER B Suggested Score: 3 

Rationale: Here is a good example of a borderline paper that will probably receive some 4s. The 
language is clear and crisp, the mechanics sound, the vocabulary above average. What holds this 
paper back is a certain choppiness; it has the tone of a "short answer" about it. Ideas are tossed 
out rapidfire with littledevelopment, and the piece ends abruptly. At the same time, B's position 
is unequivocal; there is no hedging, no hesitation. Notice that B has transposed the names Mike 
and Toby, but this is more properly a reading problem than a writing problem. 

Agree 

Disagree 

Comments: 



PAPER C Suggested Score: 1 

Rationale: C wants to avoid the whole issue and does so by neatly depositing the bird at the zoo. 
That iraght be a workable solution except that the question is not "What would yonder but "Did 
Toby make the right decisionr The response is too brief, poorly developed, and off the mark. C's 
sentence structure is only fair, vocabulary and word choice minimally adequate to answer the 
item. There is no fluidity, no sense of control. A slight change - suggesting that Toby take the 
bird to the zoo - might have earned this paper a 2. 

Agree 

Disagree 

Comments: 
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PAPER D Suggested Score: 4 

Rationale: D shows greater insight than the other papers examined thus far, presenting two 
sides to the issue, then proposing a resolution that clarifies the primary-position of the paper (i.e., 
Toby should release the bird ... "). The paper has good continuity; ideas seem to build on one an- 
other in a logical manner and transitions are smooth. (Compare Paper B.) This writer is clearly in 
control of the topic. 

Agree 

Disagree 

Comments: 



PAPER E Suggested Score: 2/3 

Rationale: This paper is difficult to score. On the one hand, writer E, based on the scant evidence 
available, is a skillful writer. E addresses the problem directly, albeit briefly, and meets the bare- 
"J? requirements of the task: taking a stand and providing a reason for that stand. On the 
other hand, there is no development, no evidence of reasoning, no personal involvement with the 
topic. This response goes beyond concise; it's terse - almost to the point of sounding flip Ifs 
tempting to allow the purpose of testing to influence the score in a case like this. For example, if 
toe purpose were to see how well a given group of students could write (i.e., as a group, not as in- 
dividuals), this student would probably receive a 2. But if students who scored below 3 were to be 
given remedial help, this student would probably receive a 3. E doesn't give us much to go on, but 
appears competent just the same. It is important to discuss this sort of issue openly with all raters 
and to agree on one approach - prior to scoring, if possible. 

Agree 

Disagree 

Comments: 



SUMMARY 

Take a moment to review your scores on papers A through E. If your scores did not differ 
by more than one point in any case, you are in virtual agreement with the scores suggested by the 
author, and are probably ready to begin applying the range finders to a larger sample and to con- 
struct exercises for classroom use. 

If you disagreed significantly (i.e., a discrepancy of two or three points) with just one 
score, there is little cause for concern. Rating papers is a complex skill; there simply are no clear- 
cut right and wrong answers. 

If you disagreed significantly in more than one case, recheck any comments you noted in 
the Rationale section; then ask yourself the following questions: 

1. Am I giving too much emphasis to mechanics? 

2. Am I automatically downgrading a paper for spelling errors without considering 
spelling in balance with other factors? 

3. Am I overly influenced by a paper's length? 

4. Am I remembering to refer to the ranger finders frequently? 

5. Am I taking too much tune to ponder each score? (Holistic scoring, remember, 
seldom requires more than 2-3 minutes per paper.) 
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Additional practice 

If you would Mke additional practice in holistic scoring, please turn to Unit 2, page 15, and 
score papers p through !. As in Unit 1, suggested scores and rationales are provided in each case. 
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UNIT 2: HOLISTIC SCORING PRACTICE (PAPERS F-I) 



Score the following student papers exactly as you scored papers A-E in Unit 1. Use the 
same set of range finders. Enter all scores before checking the suggested scores and rationales 
that appear at the end of this unit. x 

PAPER F SCORE: 

Toby has definitely not (NOT) made the right decision. Birds have feelings too. How would you 
like to be kept in a tiny cage for people to stare at and laugh at? Birds and other wild animals have 
as much right to life on this planit as anybody else. We should not kill wild animals for thier fur or 
soon they will all be dead. It is allright to help an animal that is hurt like maybe it is caught in a 
trap but then after that you, should let it go. 

PAPER G SCORE: 

I know from my own experience just how Toby feels. Once a bird flew in to our livingroom win- 
dow and was knocked unconscious. For a whole day it looked like the bird would die. When it 
came to it wouldn't eat and I had to feed it with a eye dropper. I kept it in a box in the garage. 
Then one day as I was taking it some thing to eat I discovered the box was empty! My beautiful 
pet bird had flown away never to return! At first I wished I had never seen this bird; then I would 
not have to feed so sad! But then I realized that at least I had helped the bird get well and that 
was better than having it for a petfl would of kept the bird if I could, but it would of been wrcng. 
So thinking about my own experience, Toby should let the bird go free. 

PAPER H SCORE: 

Toby has done a fine thing in caring for this helpless bird. Without his assistance, the bird would 
probably have become the prey of a fox or hawk. (What kind of bird is Mike anyway - it doesn't 
say.) But then Toby undid all the good he had done by deciding to make apet of a bird that be- 
longs free in the skies. It would have been better to let the bird die. Does that sound cruel? It isn't 
Man should not interfear with Nature. At least Toby should have known enough to release the 
bird once he had done all he could do to help it. So no, I cannot say Toby has done the right thing. 
A person who knows enough to help a bird with a broken wing should also know enough to re- 
lease it once the wing is strong again. 

PAPER I SCORE: 

Toby has made the best decision under the circumstances. Once a bird (or any wild animal) is 
tamed, it cannot be successfully returned to a wild state (with a few exceptions). Once Toby 
touched the bird, it became his responsibilaty forever. Other birds would detect the scent of man 
on the bird and would reject it. This means that Mike would never find a mate and would have 
trouble being acsepted by the other birds so he would not find refuge in a flock. Therefore, he 
would not be protected from preditors. His only hope is to have Toby continue to care for him. 
Toby is mainly keeping the bird to have a pet, but it has turned out for the best anyway. Toby can 
provide a better life for Mike than if he had, to live as an outcast. 



PLEASE TURN THE PAGE FOR SUGGESTED SCORES AND RATIONALES. 



UNIT 2: SUGGESTED SCORES AND RATIONALES (PAPERS F-I) 
PAPER F Suggested Score: 2 

This paper has a touch of originality - F manages to see things from the bird's eye view. How- 
ever, the paper is essentially lacking in control. F wanders from the issue at hand to generalize 
broadly about treatment of wild animals and never returns to the central topic. The point about 
killing animals for their fur could be related, but F does not attempt a transition; the reader has 
to bridge the gap. The shifts from first to second to third person are a bit troublesome too. Sen- 
tence structure starts out strong but rapidly disintegrates. 

PAPER G Suggested Score: 3 

This is a difficult paper to score; many factors are operating here. First, G offers a strong open- 
ing sentence - but then seems to wander off the topic. Actually, though, G uses a parallel anec- 
dote effectively - unlike F, who tosses out a seemingly related idea but never establishes the re- 
lationship (letting the idea thus become a distraction). There are problems: would of for would 
have; dangling participle in the last sentence; overuse of the exclamation point. But on the other 
' Su 0 reasonably good diction, good continuity, sound transitions, and a fresh approach. ' 
The response has clarity and style. The vocabulary is not particularly sophisticated, but the per- 
sonal touch makes the piece readable. In short, the paper is appealing, though less than excellent. 
PAPER H Suggested Score: 4* 

Good sentence structure, continuity, and consistent strong tone of conviction are the primary 
strengths here. Thera are some problems with mechanics, but they are not significant enough to 
be distracting. What really makes this paper stand out is the sense that H has thought the prob- 
lem through carefully and so is able to respond with insight and thoughtfulness. Here is a writer 
who has something to say and who says it without hesitation or wordiness. 

PAPER I Suggested Score: 4 

Notice how I presents a point of view quite different from that of most other respondents - yet 
manages to present a convincing case. Do you agree with I's logic? If not, did you find your dis- 
agreement influencing your rating? This is a common problem for raters, so a writer who adopts 
an unpopular view is sometimes at a disadvantage. I has many strengths to recommend it: good 
vocabulary and sentence structure, good diction, fine transitions. If you found this a persuasive 
paper, you likely also felt that persuasiveness enhanced the other qualities. 

Additional practice 

If you would like additional practice in holistic scoring, please turn to Unit 3, p. 17, and 
score papers GG-Jj . 

Suggested scores and rationales are provided at the end of the unit. 

NOTE: Chapters III and TV of this handbook, covering analytical and pri- 
mary trait scoring respectively, contain examples showing how a response to 
Item 1 might be scored other than hdistkaUy. 
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UNIT 3: HOLISTIC SCORING PRACTICE (PAPERS GG-JJ) 

Score the following student papers exactly as you scored papers A- 1 in the first two units 
Enter all scores before checking the suggested scores and rationales at the end of the unit. 

PAPER GG SCORE: 



What a dumb idea. What kind of a pet does a wild bird make? You can't teach it to do anything 
but just sit there. So after all the trouble you will go to then what will you have. So if you would 
ask me, then I would say, "No way"! 



PAPER HH SCORE: 

Wild birdsare very intelligent. They are hunters by nature. They will not ajust very well to life in 
a prison. That is just what a cage is, a prison. If you love an animal you want to see it happy You 
donot shut it up in a little box where it cannot move or breath. My decision is to let the bird go 

PAPER II. SCORE: 

Did Toby make the right decision? Who can say? It all depends on the kind of care he gives Mike 
If he is kind to the bird and treats it fairly and the bird is happy (this is very important!!) then it is 
alright for Toby to keep Mike a? a pet. But if he mistreats the bird, that could be considered cruel- 
ty to animals. In that case I would say he has made a very poor decision. He (Toby) must wait to 
see rt Mike makes a good pet. Birds have personalities just like people! Maybe Mike will make a 
good pet and maybe he won't. Only time will tell. 

PAPER JJ SCORE: 

Toby's decision is understandable, but that does not make it the right decision. It is hard to let go 
of something you love. But the important thing is what is the best thing for Mike? The best thing 
for any wild bird is to be free forever. So the choice is clear - Toby should let the bird go 



PLEASE TURN THE PAGE FOR SUGGESTED SCORES AND RATIONALES. 
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UNIT 3: SUGGESTED SCORES AND RATIONALES (PAPERS GG-J J) 
PAPER GG Suggested Score: 2 

G takes a defirute -in fact downright angry - stand on the issue. Unfortunately, intensity of 
feeling isn't enough to lift this paper out of mediocrity. Sentence structure and vocabulary are 
weak; the style is arresting, perhaps, but scarcely imaginative, logical or entertaining. 

Agree 

Disagree 

Comments: 



PAPER HH Suggested Score: 3 

Here is a brief but well organized statement in favor of freeing Mike. The writer provides sound 
arguments: birds are intelligent, they're hunters, they can't adjust ts^ife in a cage. The sentence 
structure is adequate, but repetitious and choppy. Some variationlhitructure and rhythm could 
have earned this paper a 4. 

, Agree 

Disagree 

Comments: 

PAPER II Suggested Score* 2/3 

How you score this paper probably depends a great deal on whether you think I is avoiding the is- 
sue or just carefully considering both sides while awaiting further details. I is saying, in effect, "I 
need more information to give you my final decision." The paper shows thought and imagination. 
It offers a good attention-getting opening, too, though the cliche ending is a disappointment. In 
addition, I switches topics: whether Mike makes a good pet is a quite different issue from whether 
Toby treats Mike fairly. This shift is poorly handled and weakens an otherwise fair effort. 

Agree 

. Disagree * 

Comments: 

\ 

PAPER JJ Suggested Score: 4 

Good, varied sentence structure and simple yet highly appropriate vocabulary make this response 
readable and pleasing to the ear. There's a punctuation problem in the third sentence, but it's 
minor. "Free forever* has a nice ring. ^ 

Agree 

Disagree 

Comments: 



THIS CONCLUDES THE CHAPTER ON HOLISTIC SCORING; THE FOLLOWING 
CHAPTER OFFERS PRACTICE IN ANALYTICAL SCORING. 
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CHAPTER in: Anatytical Scoring 

Practice 



Lake Chapter II, this chapter contains a sample test item and hypothetical student re- 
sponses to that item. This time you will be asked to score the student responses analytically then 
- as before - to compare your scores with those suggested by the author. Given practice and a 
chance to compare your reasoning with the 'rationales provided, it is anticipated that you will 
achieve a high level of agreement with the author's scores. Again a reminder that occasional dis- 
agreement results from individual approaches to scoring and should be no real cause for concern 
Consistent disagreement however, should p#mpt you to carefully reexamine your thinkme in 
light of some guidelines that the author will suggest. • * 

HOW TO SCORE 

Analytical scoring - quite unlike holistic scoring - is a trait-by-trait analysis of a paper's 
merits. It is not impressionistic as holistic is; rather, it calls for indepth review. The individual 
traits scored (usually there are five or six) are those considered important to any piece of writing 
m any context In other words, they are traits that would be important whether one were writine 
an essay on endangered wildlife or a letter of complaint to a local department store. 

Three trait*. Item 2, (page 20), calls for students to write a letter taking a firm itand in re- 

S^SL? T^L 1°!^. £ e Ietters on ^ traits: ideas. organization and 

wm^. Tt^are three of the five significant traits that researcher Paul Diederich isolated 
through his study. In an actual analytical scoring session, you would probably be asked to deal 
with five or six traits in order to get a complete picture of each paper's merit. For purposeTof this 
practice, three traits are ifficient, F ^ 

The "halo" effect.. Traits should be scored one at a time. In scoring, take care not to let 
your impression of one trait (say, ideas) influence your scoring of another trait (say, organiza- 
tion). Sometimes raters fall victim to what is termed the "halo" effect: rating all traits on the 
same level. In other words; a paper with outstanding ideas may be perceived as outstanding in all 
respects - when in fact that may not be the case at all. A writer may have truly original ideas but 
organizeHhem poorly, while another has sound organization but unimaginative wording, and so 

A eftterton baaed sysUm. As you-have seen,-]iq}istic scoring is a normative ranking of pa- 
pers, in wnftji scores defive meaning on the basis of how each paper compares to «>11 others in a 
given sample. Analytical scoring is quite different. It is criterion based. In other words, raters are 
given specific guidelines, tilings to look for- in assigning scores. How one paper compares to 
others is of no real importance ill analytical scoring. It's how that paper measures up to specific 



J^^"^ experienced scorers trained in holistic procedures to specify which qualities of writing influenced their 
-scores most Through systematic analysis of the results, Diederich isolated five factors whiCh scorers seem to focus on in 
rating, wpen: ideas, mechanics, organization, wording and flavor (or what some people term "style"). See Diederich. 
Paul B. Maturing Growth in Englith. Urbana, IL: National Council of teachers of English, 1974. 

S 5 ™" rneasum 6/wriHng skill- Issues and application. Portland, OR: North- 
west Regional Educational Laboratory, 1960. 
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criteria that counts. For this reason, range finder papers are not generally used. Instead, raters 
are g*ven written guidelines to aid them in scoring. 

' The guidelines. The following pages present guidelines for scoring responses to Item 2. 
Remember that you will score each response three times: for ideas, wording and organization. 
The final score for each paper will be a total of these three subscores. In other words, possible 
total scores will range from three (one point for each factor) to 12 (four points for each factor). 

The definitions and criteria descriptors presented in these guidelines were developed 
specifically for use with this test item. While the factors to be scored (i.e., ideas, wording and 
organization) have universal application, it is likely that these definitions would require modifica- 
tion prior to use with a different test item. 

Use these guidelines with flexibility. It is not necessary that & given paper conform totally 
to the definition presented* Every paper is unique and not even the broadest guidelines can ac- 
count for every variable* Therefore, assign scores according to which definition seems to best fit 
a given paper. 

When You Ye Ready to Score ? 

To begin, read through Item 2. Then read the definition and criteria fot scoring carefully. 
It is recommended that you remove the guidelines for scoring Item 2 from your notebook and 
keep them in front of you so you can refer to them as often as you want. Before scoring you may 
also wish to lode over the two sample papers scored for you at the beginning of the unit. 

Score all five papers for Unit 4 (K-O) before turning to the suggested scores and Irationales, 
page 28. 



ANALYTICAL SCORING: Sample Test Item* 

Following is a sample test item that might be given to a group of ninth graders to measure 
competency in formulating wording and organizing ideas. You can assume that students 
would be given as much time as they needed to respond, and would be allowed to make correc- 
tions on their papers if they wished. Also assume that instructions would be given orally as well 
as in writing. 



Item 2 

Exercise: Letter to the Chairman of the School Board 
Directions to the Student: 

Imagine that your local school board is considering adopting a resolution to 
hold school year-round. The opinions of everyone involved - including 
teachers, parents and students - will be considered. You have been asked, 
therefore, to write a letter to John Brown, Chairman of the School Board, 
stating your views. 

In your letter, state clearly whether you are for or against the 12-month 
school year. Give reasons for your opinion. Be specific. Use examples from 
your own experience to support your answer. Sign your letter Tat Smith." 



'Items in th» handbook are considered adaptable for use with any grade level, 
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ANALYTICAL SCORING GUIDE 
Factor 1: IDEAS 



DEFINITION: 

CRITERIA: 
4 Paper: 



3 Paper: 



2 Paper: 



1 Paper: 



For purposes of this test item, ideas may be defined as arguments or opi- 
nions, or any thoughts expressed in support of those arguments/opinions. 



A 4 paper offers sound ideas that indicate the writer has given careful 
consideration to both sides of the argument. 

Common characteristics:* 
•Originality 

• Logic, not emotion 

• Insight or h"mor 

A 3 paper resembles a 4; the ideas are simply not as sound or as s f riking. 

Common characteristics: 

• Clarity 

• Less insight than 4 

• More emotion than 4 (less logic) 

A 2 paper offers little evidence of careful judgment or insight. 

Common characteristics: 

• Arbitrary/whunsical ideas 

• Emotion, not logic 

• Lack of clarity 

• Lack of originality 

In a 1 paper, clarity may be such a problem that the reader can scarcely 
discern the writer's viewpoint. 

Common characteristics: 

• Ideas just a restatement of the question 

• Lack of insight/originality 

• Lack of clarity 

• Ambiguity 

• Indecipherable ideas 



'^S^ISS^S^. to at ** ,evels Mcato,: however - - ^ s 



so 
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Factor 2: ORGANIZATION 



DEFINITION: 



CRITERIA: 
4 Paper: 



3 Paper: 



2 Paper: 



1 Paper: 



For purposes of this test item, organization may be defined as a sense of 
order, ability to stay on topic, anchability to relate all details to a central 
idea or argument 

The 4 paper is carefully organized from beginning to end. 

Common characteristics: 

• Main points clear 

• No contradictions 

• No shifts in viewpoint 

• Good supporting evidence 

• Logical order 

• Noth&fc omitted 

• Writer stays on target 

Hie 3 paper is generally well organized, though there may be some lapses. 

Common characteristics: 

• Main points can be inferred 

• Writer takes a definite stand 

• Support may be weak 

• Some irrelevancies 

The 2 paper shows some attempt at organization, but the result is 
ineffective. 

Common characteristics: ^ 

• Absence of clear stand 

• Weak/ambiguous support 

• Important details omitted 

• Shifts in viewpoint common 

• Details presented in random/haphazard order 

The 1 paper may be so short that it defies organization, or so muddled that 
the reader can discern no reasonable pattern of thought. 

Common characteristics: 

• Writer's viewpoint unclear 

• Lack of support 

• Circular arguments 

• Reliance on opinions, not logic 

• Total lack of development 



Factor 3: WORDING 



DEFINITION: 

CRITERIA: 
4 Paper: 



3 Paper: 



2 Paper: 



1 Paper: 



For the purposes of this test item, wording may be defined as vocabulary 
and word choice. 



A 4 paper exhibits strong, carefully chosen vocabulary. 

Common characteristics: 

• Correct usage 

• Word choice appropriate to topic 

• Vocabulary effective 

• Absence of cliches, jargon, slang 
•Lang n ge fresh, original 

A 3 paper exhibits acceptable, comprehensible vocabulary. 

Common characteristics: 

• Generally correct usage 

• Word choice appropriate, not dramatic 

• Slang and jargon minimal 



A 2 paper typically contains numerous errors in usage and word choice. 



Common characteristics: 

• Simplistic vocabulary 

• Errors in usage 

• Poor readability 

• Reliance on slang/jargon 

• No spontaneity/flow 

A 1 paper is nearly impossible to decipher. 

Common characteristics: 

• Word choice poor 

• Faulty usage 

• Less than rudimentary vocabulary 

• Serious errors that impair readability 
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Scoring the Sample Test Item 



Sample Paper 1 Ideas: 2_ 

Organization: 2/3 

Wording: 2_ 

Total: 6/7 



Dear Chairman Brown, 

What can you be thinking of?! Who needs 12 months of school? Not us!! 

Soon enough we will be part of the work forse and then our days of fun will be over for good. 

Why should we have to give up the one good times we have which is our summ§rs off. I only 
hope others will feel like me and will write to you so this idea will not go through because it 
would be a big, BIG mistake. 

Bye and thanks you, 
Pat Smith 

Rationale: It is simple to discern the writer's viewpoint; however, ideas are expressed in 
largely emotional terms with little evidence of insight or originality. Wording is clear, despite 
some awkward phrasing, but certainly simplistic. The writer states a clear position and does 
not wander from the main topic, but supporting "evidence" is sketchy at best. The paper is 
highly redundant, and offers little save reiteration of the writer's opinion. 



Sample Paper 2: Ideas: 3 

Organization: : 3 

Wording: 3 

Total: 9 



Hello Chairperson Brown: 

How can I convince you to abandon the idea of the year-roupd school year? Maybe if you had 
ever tried it you would feel differently. ' " ' 

Last year I attended summer school and I found out what it is like to go to school all year with- 
out any time off. It is tiring! Not even grownups and others who work nave to work au the time. 
We could come down with severe eye strain from studying too hard without any breaks. 

I have met you once before during the school assembly and I know you are a friendly and rea- 
sonable person. If anyone can make the right decision it will be you. 

Your friend, 
Pat Smith 



Sri 
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Rationale: The central argument here seems to be that year-round school is tiring. This idea is 
har^y original but it is adequately and clearly expressed. The writer is unquestionably opposed 
« 2 f Jt > schoo iy ear - ™ d never veers from that stand. There is a problem with transi- 
tions, the writer goes from point to noint without attempting smooth connections, and the last 
paragraph is unrelated - but that doesn't damage continuity since it comes at the end anyhow. 
Wording ib clear and satisfactory though far from inspired. Greater development of ideas and 
good transitions would have pushed this paper toward 4s. 

UNIT 4: ANALYTICAL SCORING PRACTICE (PAPERS K-O) 

„ 0 w K?* 4 .? c 1 ! udes fi ve .Papers, K-O. Score each paper analytically on each of three traits, 
using the guidelines provided. Score each trait on a 4-point scale. Enter your scores in the 
spaces indicated. It is strongly recommended that you score all five papers for ideas first, then 
go back and score all for organization, then all for wording. Dealing with the traits one at a time 
is less confusing, and will help you to internalize the criteria. 

t , ™™J m ^ a? 01 * 1 ^ five Papers on all three traits, check your scores against 
the suggested scores and rationales provided by the author, page 28. 

Scoring will go faster and be easier if you are thoroughly familiar with the Scoring Guide 
U often dir^sclri^ 11 " 16 * ** ^ tl " 0Ugh b * ta * SCorin *' «"* ^ fer to 

Analytical scoring generally takes 2-5 minutes per trait, or 6-15 minutes per 3-trait pa- 
per. If you find yourself ^consistently taking longer, you may be deliberating too much, or you 
may not be sufficiently familiar with the Scoring Glide. «n«4i,wyuu 

NOTE: If for any treason you think ym may wish to reuse these rnateriats, 
or share them with a colleague, we suggest you 'make copies of student re- 
sponses before scoring. 

Sample Scoring Exercises for Unit 4 (Papers K-O) 

Enter your scores in the blanks provided. \ 
PAPER K Ideas: 

Organization: % 

Wording: 

Total: 

Dear Chairman Brown, 

S^^fi^^ f 6 ^-month school year. A lot of administrative paperwork could 
be reduced if schools did not have to shut down and reopen each year. More time spent in school 

5^?^iL^ tter ^ U ? ltk)n f0 , r aU students - TtoTwhy, you may ask, dofopp^f Z 
12-month school year? Let me explain my reasons. 

First, a young perron's experience should not be restricted to school. We all need variety to 

SSTlC 7Hu ST 8 ' k We * mUSt 5 ttend ^A 001 12 L months out of the year our imagination will stag- 
nate hke old slough water and we will be nothing more than programmed robots. 

Second, your idea un<termines i family closeness. At first parents may find it exciting to be free 
n2T™ n<J WnsibUi^ for their children But as years pass id there is no more dose 
ness or companionship in the family, parents will regret this decision. By then it will be too late 
for school will not just fill our lives, out it will be our lives. Familes will come second 

Chairman Brown, you must think of your constituences. Do not act in haste. 

Sincerely yours, 
Pat Smith 
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PAPER L Ideu: 

Organization: 

Wording: 

Total: 

Dear Chairman Brown 

Keep the schools open. No way! This is a crummy idea and it is again our personal right So if 
you can open the school and then you are aksing to have a rebelion on you hands That will be 
your felt. 

From Pat Smith 



PAPER M Ideas: 

Organization: 

Wording: 

Total: < 

Dear Chairman: 

The 12 month school year you suggest presents a problem. Alot of kids will not mind going to 
school all year because it will get them off the streets and give them something to do. lam not 
one of them. Because have you thought about the hundreds of kids who depend on summer 
jobs. For them it will mean a big loss of income. Parents, I think will like your idea and they 
should have a say in making this rule. I don't want to give you a bad time but your idea won't be 
a big hit with most of the kids. Most kids do not enjoy school all that much. I myself am planning 
to work this summer so this will be hard for me. Maybe you could think about making summer 
vacation shorter but not as much. Thanks. 

Pat Smith 



PAPER N Ideas: 

Organization: 

w ording: 

Total: 

Dear BJr. Brown, 

Your thinking has caused a dilemma. You have come up with the idea that is good but that no- 
body will like. I hav the solushion to you problem. 

Keep the 12 month school year but do not mak every school year twelve months long. By that I 
mean to let students go through at their own rate so if ninth grade taks 12 months mat is O.K. 
or if it taks some kids only five months that is fine too and then they would go right on to 10th 
grade with no time wasted. This way the kids could finish school as fast as they wanted and go 
on to other stuff. For the slow kids it would be a good idea, they would be in school all the time 
and would not have so much time to forget things like over the summer holliday. So I say to 
keep vour idea about the 12th month school, jt is a good idea only do not mak the fast kids and 
the slow kids go to school in the same amount of time but let the fast kids go through faster. 

Sincerly. 
Julie Baker 
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PAPER O Ideas: 

Organization: 

Wording: 

Total: 

Dear John Brown, 

As Chairman of the Board I am askingyou to forget the 12 month year it will only lead to prob- 
lems. Think first of the energy crissis. If we have school all the time what will it take to heat the 
buildings and classrooms. Our heat and lights will be shut off then every one will suffer. Their 
. areother way. Like you would give us more vacashjons only shorter so it would still add up to 3 
months. Grye us more time at Christmass when its cold and not so much time in the summer 
when it is hot. But do not take away all our privleges, ok. 

"Sincerely, - s 
Pat K. Smith 



PLEASE TURN THE PAGE FOR SUGGESTED SCORES AND RATIONALES 



V J 
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UNIT 4: SUGGESTED SCORES AND RATIONALES (PAPERS K-O) 
PAPER K Suggested Scores 



Ideas: 4 

Organization: 4_ 

Wording: 4 



Agree Disagree* 



Total: 12 



Rationale: This & a superior paper in all three respects. K has considered both sides of the ar- 
gument, then taken a clear stand in opposition. K's arguments are imaginative, forceful and col- 
orfully expressed ( stagnate like old slough water"). The vocabulary is rich but not stuffy; it en- 
hances readability. Ideas are carefully paragraphed and logically ordered. A minor spelling er- 
ror ( constituences^, but no real problems; K is a writer in control. 

(Space is provided here for any comments you wish to make for your own future reference.) 
.Comments: 



PAPER L Suggested Scores 
Ideas: 

Organization: 

Wording: 

Total: 



Agree Disagree 



Rationale: Here is a highly emotional response to the issue, with little evidence of careful 
thought. The "ideas" are little more than the writer's opinion, though certainly L's stand is clear 
(despite punctuation problems in the first two sentences). The paper lacks any sense of organi- 
zation; it is too short, and provides no supportive evidence for its only argument: "This is a 
crummy idea." Vocabulary is weak to be sure; however, it cannot truly be said that L's meaning 
is undecipherable. One senses that L would have a good deal more to say given a more 
sophisticated vocabulary. 

Comments: 



•Remember that a one-point discrepancy is not considered true disagreement. Only discrepancies of two or three 
points are judged disagreements. 
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PAPER M Suggested Scores 
Ideas: 

Organization: 

Wording: 

Total: 



Agree Disagree 



Rationale: What a jumble of ideas M*s paper is! The 12-month school year, says M, will get kids 
off the streets and "give them something to do* - yet later M reneges a bit, suggesting that the 
Idea won't be a big hit with most of the kids." In between, M wanders from the topic completely 
to introduce the subject of parent involvement, an issue never explored or developed. At the 
end, M injects an afterthought about shortening summer vacation. While one can reasonably in- 
fer M*s opposition, the thinking overall is disjointed, the wording simplistic. This is a paper that 
opens on a positive note but goes nowhere. 

Comments: 



Agree Disagree 

3 

3 

2 

8 



Rationale: N's idea reflects some originality: allowing students to go through school at their 
own rate, while retaining the 12-month school year. Unfortunately, the paper is so wordy and 
redundant, the originality is virtually buried. The wording is clumsy and ineffective. On the 
other hand, the paper is reasonably well organised. The first paragraph introduces the solution, 
the second presents it in orderly - albeit tedious - fashion. Simple condensation and snappy 
vocabulary could have rescued this mundane response. 

NOTE: Mechanics are pqpr in this paper. Did you find that factor influence 
ing your scoring of other factors? Remember the importance of scoring each 
factor separately. 

Comments: 



PAPER N Suggested Scores 
Ideas: 

Organization: 

Wording: 

Total: 




3G 
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PAPEH 0 Suggested Scores 
Ideas: 

Organization: 

Wording: 

Total: 



Agree Disagree 



• Rationale: 0 opens with a misplaced modifier and never quite gets back on track "Think first 
of our energy crisas" [sic] sounds hopeful, but 0 never gets beyond the energy crisis, so the pa- 
per dwindles to a one-point argument - and is thus weakened considerably. This paper (like 
Nty suffers most from ineffective wording. This writer is not in control; in feet, the letter 
seems laborious and awkward. Still, 0 takes a definite stand and sticks by it. The letter is not 
hard to follow - it's just dull reading. 

Comments: 



4 



SUMMARY 

Take a moment to review your scores on papers K through 0. If your scores did not differ 
Iw more than one point (three points for "totals") in any case, you are in virtual agreement with 
the scores suggested by the author, and are probably ready to begin incorporating analytical 
scoring procedures into classroom instruction. 

K you (iisagreed significantly (i.e., a discrepancy of two or three points) with just one 
score, there is little cause for concern. No two raters will ever agree fully on any given set of 

If you disagreed significantly in more than one case, however, recheck any comments you 
noted in the Rationale section; then ask yourself the following questions: 

1. Am I allowing poor mechanics to unduly influence my scoring of other factors? 

2. Am I referring to the Scoring Guide (i.e., criteria) frequently? 

3. Am I remembering to rate each factor independently, regardless of how well or 
how poorly a student does with respect to other factors? 

4. Am I taking time to read each letter through or am I slamming too quickly? 

NOTE: It's tempting to rush through a particularly dull piece; remember, excep- 
tional idea* may be camouflaged by poor wording. 

5. Am I allowing the tone of the letter to unduly influence my judgment? 

NOTE: The most courteous letter may or may not be the best; it may indeed have 
little to commend it but courtesy. 

Additional practice 

If you would like additional practice with analytical scoring, please turn to Unit 5, page 31 
and score papers P through T. As with Unit 4, suggested scores and rationales are provided 
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UNIT 5: ANALYTICAL SCORING PRACTICE (PAPERS P-T) 

Score the following student papers exactly as you scored papers K-0 in Unit 4. Use the 
same goring guidelines. Enter all subscores and totals before checking the suggested scores 
and rationales at the end of this unit. 

PAPER P Idea*: 

Organization: 

Wording: 

Total: 

Dear Chairman Brown: 

Who says we should have school for 12 months out of the year? That person should have their 
head examined. Its one thing to keep us going to school all during the winter when it is raining 
and cold and their is not that much stuff going on anyway. But now we are suposed to just say 
oh to heck with summer vacation and keep right on going to school like we had nothing better 
to do? Does this make sense to you? No reasonable person would vote for this measure. 

Pat Smith 

PAPER Q Ideas: 

Organization: 

Wording: ' 

Total: 



Dear Chairman Brown, 

I am writing this letter to inform you that I strongly oppose the 12-month school year for three 
reasons. . 

First, students need an occasional break in order to do their best work. JVe return from sum- 
mer vacation refreshed and invigorated, ready for the year ahead. If wei^ist work the year 
around, many of us will grow bored and lose interest in school. I predict the drop-out rate will 
increase substantially. 

Second, teachers need a break too. Would you like to face 30 years in the classroom with only a 
day off here and there? It's inhuman. Our teachers need time to plan lessons and think about 
the curriculum for the coming year. This takes time and energy. Without summer vacation 
they will have neither one. 

And last, many students need to work during summer vacation. They need the money. You can 
not get a job anywhere with only two weeks off. Will you deprive students of this important op- 
portunity? 

Please consider the points I have made in this letter and present them to your Board. Your deci- 
sion effects the lives of many people. I hope you make the right choice. 
Sincerely, ' * 

Pat Smith 



ERLC 
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PAPER R Ideas: 

Organization: 

Wording: 

Total: « 

Dear Mr. Chairman Brown, 

I got your letter about the 12-months school year and I am inraged. How can you be so short 
sited? Do you think we will just stand by and let you take all the fun out of our lives? We need 
our vacations td rest up or we will die! We are too young to be slavs. No matter how important a 
good education may be. We are not saying it is not important what we are saying is for you to 
think about it from our side. Will we learn, more if we go more? No! It is useles to forse us 
against our wills. 

Sincerely your 
Pat Smith 



PAPERS 



Ideas: 

Organisation: 

Wording: 

Total: 



Dear 

Thank you j 
to you. It is 
could spend 
make our liv 
gets rid of v 
only questi 
Thank you 
Pat Smith 



Brown, 



for/ei 



smerring with this supurb idea. Many of your students will be imensely grateful 
belief that too many students waste the time known as "summer vacation." We 
_s tame more ^profitabally attending school and learning something that would 
easier. The 12-month school year is the perfect solution to y ear-around school. It 
ted summer time and lengthens our learning opportunities at the same time. My 
to you is why didnt someone think of this sooner and save us all alot of erie* 
— Chairman Brown - 



/ 



PAPER T 



Ideas: 

Organisation: 

Wording: 

Total: 




Dear Chairman Brown, 

I personally like your idea a lot, but it won't work. Here's why. 

S3 722 m it; ^ 8° «J 0| W f or * <* two, but then they will get pretty 

bored and want their summer vacation back. 

Sf?^ te f9 her8 Men. Alot of teachers become teachers because this is a job 

that offers good hours. Now you are taking that benefit away. Be careful! Soon no one will 
want to be a teacher and then there will be no one to run the schools. 
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Thirdly, you are going to get parents angry. Their vacation plans will be ruined if summer vaca- 
tion for kids is taken away. 

Can you afford to have all these people mad at you? You #ill have no friends left! So . . . think it 
over and then make up your mind. We are all waiting for your decision. 
Sincerely, 
Pat Smith 

PLEASE TURN THE PAGE FOR SUGGESTED SCORES AND RATIONALES. 



10 
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UNIT 5: SUGGESTED SCORES AND RATIONALES (P-T) 
PAPER P Suggested Scores 

Ideas: , 2 

Organization: 2 



Afrr agree 



Wording: 1/2 



Total: 5/6 



T 



* 9 

Rationale: This paper is, to say the least, limited in scope. P argues that the 12-month school 
year is a bad idea because he/she doesn't like it - an argument that is neither insightful nor strik- 
ing; The language is unsophisticated and flawed (e.g., "That person should have fyeir head ex- 
amined"). As brief as this response is, if s wordy, given its content. Organization is difficult to 
iicore on such a paper. One can, certainly, follow the argument; there's no ambiguity. On th6 
other hand, what is there to follow? P basically dislikes school and hopes the reader will join in 
keeping the school year as short as possible - not persuasive. 
Comments: 



PAPER Q Suggested Scores 

Agree Disagree 
Ideas: 4 { 

Organization; 4 

Wording: 4 ~~ 



Total: 12 



Rationale: Here's a letter that shows some thought. The ideas are clearly *exr ressed and go 
beyond the writer's own biases: increased dropout rates, imposition <#v teachers' time, decreased 
owxatunities for students to work. The wording is appropriate to the topic; however, if you felt 
the formal tore was a little forced in spots, you may have scored Q's letter 3 in this regard. With 
respect to organization, this paper is a classic: an intro, three point-by-ppint paragraphs, and a 
nifty wrap-up. Who could ask for more? 
Comments: > 



41 



PAPER R 



Suggested Scores 



1 



Ideas: 

Organization: 

Wording: 

Total: 5/6 



2/3 



Agree Disagree 



Rationale: R's approach is a little different. "I got your better . . ' What letter? R doesn't have a 
lot to say: you shouldn't make slaves of us. Thafs about it. Nevertheless, the wording has a cer- 
tain charm; if s forceful, to the point, unquestionably clear. The writer manages to inject emotion 
without coming across as angry (despite the claim to be "inraged" [sic]). It's pretty tough to 
organize a paper that doesn't say much, and indeed R's letter breaks down here. A touch < 



elevates this letter above total disaster, but can't save it 
Comments: 



i of charm 



PAPER S Suggested Scores 
Ideas: 

Organization: 

Wording: 

Total: 



Agree Disagree 



1/2 



7/8 



Rationale: Watcfe out for the halo effect on this one. S's letter is worded so well, it seems to be an 
excellent piece. But take a closer look. It's actually a clever bit of redundancy that repeats the 
same argument in a variety of ways: i.e., that summer vacation is a waste of time. (One may also 
question S's sincerity, but that's another issue altogether.) And what about this bit of nonlogic: 
TTie 12-month school year is the perfect solution to year-around school." This circular nonsense is 
almost "political'' in its evasiveness. Suppose you were Chairman Brown. Would you be charmed 
by the ingratiating tone, or put off by the lack of insight? Your persona^/esponse to S's style will 
no doubt affect your scoring. 
Comments: 
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PAPER T Suggested Scores 
Ideas: 

Organization: 

Wording: 

Total: 



Agree Disagree 



Rationale: T has done the impossible: organized a virtually content-free paper. Compare letter Q, 
which » similar in format, but gives us so much more. Ts letter is deceptive. At first glance it 
seems to be a powerhouse of ideas. In fact, though, it all comes down to this: an unpopular idea is 
f P 00 ™**- Wording is unimaginative, but certainly adequate. The personal tone adds appeal too 
(e.g., "Be careful! "You will have no friends left!"). But as we saw with paper R, charm without 
insight can t carry the day. 

Comments: 



^ Q S EL H T ^A?F SP0NSE 10 TTEM 1 MIGHT BE SCORED ANALYTICALLY (VER- 
SUS HOLISTICALLY), PLEASE TURN THE PAGE, 



43 



ERIC 



COMPARING SCORING APPROACHES: HOLISTIC vs. ANALYTICAL 

How would a paper scored holistically stand up to analytical scoring? Let's take a look. 
Following are two papers from Chapter II on holistic scoring; the original holistic scores are 
shown here for comparative purposes. 

In addition, these responses have been scored according to the Analytical Scoring Guide 
presented in Chapter III. Readers are asked to keep in mffiH that these guidelines were developed 
for use with Item 2, and are used unmodified with Item 1 for illustrative purposes only. It is 
always best to develop scoring guidelines in conjunction with the item for which they are intend- 
ed. (For the reader's convenience, Item 1 is reprinted here.) 

ITEM 1 (Reprinted from page 8 ) 

Toby loved to hike in the woods. One day when he was hiking, he found a bird with an injured 
wing. Toby took the bird home, set his wing, and took care of him. He named the bird Mike. Soon 
Mike got better, and he could have flown away, but Toby kept him in a small cage. Toby knew the 
bird was ready to be free, but he liked Mike so much he decided to keep him as a pet. 

Do you think Toby made the right decision? Write out your answer on this page. State clearly 
whether you think it was a good idea for Toby to keep Mike. Then explain your reasons as com- 
pletely as you can. 

PAPER A (Reprinted from page 11 ) 

We should try to see things from Toby's side. I mean, it isn't every day that you get a chance to 
study nature up close. Now Toby has found this wild bird and he has a chance to study it and 
observe it. Maybe it is not the best thing to happen for a bird to be kept in a cage, but how else is 
he going to learn from it. After a while, he can let the bird go. 



SUGGESTED SCORES 

Holistic: 2 Analytical: 

(See p. 12 for rationale.) Ideas: L 

Organization: 1/2 

Wording: 2_ 

Total: 5/6 



Rationale: A came up with an original idea, but didn't take it anywhere. What about studying 
the bird up close? A shifts quite arbitrarily to the ill effects of keeping a bird caged, then ends on 
a weak, indecisive note that leaves the reader dangling. Lack of direction and pcor develop- 
ment thwart any efforts at organization. Wording is barely adequate, certainly far from in- 
spired. Overall, the response degenerates from promising to dull. 
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PAPER D (Reprinted from page 11 ) 

Toby faces a dilemma. If he keeps the bird, he is depriving a wild creature of its freedom. If he 
lets the bird go, he is taking a chance that the bird will no longer be able to care for itself in the 
wild. Once tamed, a wild creature often becomes dependent. Toby should release the bird, but 
watch him carefully for several days to be sure he can care for himself. Then he will know he 
has made the right decision. His decision to keep the bird as a pet is not a responsible one and 
I'm sure that when he reconsiders he will change his mind. 

SUGGESTED SCORES 

Holistic: 4 Analytical: 

(See p. 13 for rationale.) * det8: — i — 

Organization: 4 

Wording: 4 



Total: 12 



Rationale: D has considered both sides of the argument and takes a clear, strong stand - no 
hedging. The response is concise, the points well ordered. The vocabulary is sound, the rhythm 
easy on the ear. Ua easy going style is as natural as it is articulate. Seldom does a writer com- 
press intro, main argument and viable conclusion into so short a space. This is a fine per- 
formance. 

SUMMARY 

You may wish to score other, papers from Units 1, 2 and 3 analytically, and compare 
results with the holistic scores. (For comparative purposes, a holistic 4 L roughly equivalent to 
an analytical 10-12, a 3 to an analytical 7-9, a 2 to an analytical 4-6, and a 1 to a 3.) This com- 
parative scoring is a worthwhile classroom exercise. To elicit some lively discussion, try a com- 
parative holistic scoring of a paper like T (page 46), in which the analytical subscores differ. 
Does analytical scoring provide significantly more information than holistic scoring? 

THIS CONCLUDES THE CHAPTER ON ANALYTICAL SCORING. THE FOLLOWING 
CHAPTER OFFERS PRACTICE IN PRIMARY TRAIT SCORING. 
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CHAPTER IV: Primary Trait Scoring 

Practice 



This chapter is based on the same sample test item used in Chapter III: Analytical Scor- 
ing Practice. You will, however, be asked to score papers according to different criteria from 
those you previously applied. As before, you will score a series of papers, then have a chance to 
check your scores against those suggested by the author.. As with the holistic and analytical 
practice, a high level of agreement is anticipated. 

HOW TO SCORE 

Primary trait scoring is based on the. premise that different traits are important to dif- 
ferent pieces of writing. For example, in a set of instructions for assembling a bicycle, the most 
important (that is, primary) trait may be clarity, for unless the instructions are clear and unam- 
biguous, it is unlikely the reader will be able to put the bike together. Another trait - say, tone 
- may be relatively unimportant in a set of bicycle assembly instructions, but may be critical in 
a department store's letter of apology to a disgruntled customer. 

The primary trait approach assumes that everything is written to an audience, and that 
good writing will have the desired effect upon that audience. For example, a good letter of ap- 
plication will get the job interview, a good campaign speech will win votes. 

As you recall from Chapter III, Item 2 asks "Pat Smith" to write a letter to the Chairman 
of the School Board, commenting on the advisability of a 12-month school year. The best let- 
ters, presumably, will be those that persuade Chairman Brown to adopt the writer's point of 
view. In other words, the primary trait in this case is persuasiveness. For purposes of scoring, 
you are asked to put yourself as reader in Chairman Brown's place - to pretend, if you will, 
that you are Chairman Brown. Accordingly, those letters that move you, that persuade you 
most, will receive the highest scores. Letters that are not ^persuasive cannot receive a high 
score, regardless of how well written they may be in other respects (e.g., in terms of word 
choice, sentence structure, organization). ; * 

Like analytical scoring, primary trait scoring is a criterion based - not a normative 
based - system. Therefore, scoring guidelines, rather than range finders, are used in assigning 
scores. 

When you're ready to score 

On the following page is a Guide for assigning primary trait scores on a 4-point scale. 
(Note also the definition of the special "zero category," which is introduced in this chapter.) 
Read the Guide carefully. Scoring will go faster if you are thoroughly familiar with criteria. It is 
recommended that you remove the Guide from your notebook and keep it in front of you while 
scoring so that you can refer to it often. When you have finished studying the Scoring Guide, 
take a few moments to review Item 2, page 42. Before scoring, you may also wish to look over 
the two sample papers scored for you at the beginning of this unit. 
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PRIMARY TRAIT SCORING GUIDE 



CRITERIA: 
4 Paper: 



Highly Persuasive 



The writer takes a strong stand either pro or con and defends it with solid 
evidence. 

Common characteristics: 

• Arguments logical 

• Arguments backed by fact/experience 

• Rhetorical excellence 

• Impressive style 

NOTE: The reader may agree or disagree utith the writer's paint of view; 
this is irrelevant and must not be allowed to affect sewing. A 
reader may, for example, assign a paper a 4 on the basis that 
'This was a mil written letter conveying a tound and highly per- 
suasive argument - even though it does not express the position I 
would have taken. 99 



The writer's viewpoint is clear, but supporting evidence may be less effec- 
tively presented than in a 4 paper. 

Common characteristics: 

• Arguments logical 

• Awareness of audience 

• Adequate wording, but no real flair 

• Supporting evidence may Jbe flawed or weak 

• Less rhetorically effective than a 4 



A 2 paper overall lacks the power to persuade. 
Common characteristics: 

• Viewpoint not clearly stated, but can be inferred 

• Support weak/irrelevant 

• Poor awareness of audience 

• Inappropriate tone 

• Reliance oh emotion, not topic 

• Poor word choice 



3 Paper: 



Generally Persuasive 



2 Paper: 



Generally Unpersnasive 



47 



Highly Unpersuuive 

Unlike a 2 papet, a 1 paper may be virtually incomprehensible or may ex- 
hibit no effort to persuade the reader. 

Common characteristics: 

• Viewpoint hard to determine 

• Lack of support 

• No awareness of audience 

• Inappropriate tone 

• Poor word choice 

• Reliance on emotion, not logic 

• Less continuity than a 2 

"Zero" is a special category/ It is not an indicator of quality (no paper is 
qualitatively below the 1 level). Zero should be assigned to papers that are 
Wank, totally illegible, written in a language other than English, or that 
address a completely different topic (e,g., "My Dog Fred"). 



category," though first introduced in this chapter, is often used in holistic and analytical scoring systems 



if 
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PRIMARY TRAIT SCORING: SAMPLE TEST ITEM 

For your convenience, Item 2 is repeated here. Note that students are explicitly directed to 
make the letter as persuasive as they can. As before, you can assume that students would be 
given as much time as they needed to respond, and that they would have a chance to make correc- 
tions on their papers if they wished. Also assume that instructions would be given orally as well 
as in writing. Respondents are ninth graders. 

Item 2 < 

Exercise: Letter to the Chairman of the School Board 
Directions to the Student: 

Imagine that your local school board is considering adopting a resolution to hold school 
year-round. The opinions of everyone involved - including teachers, parents and 
students - will be considered. You have been asked, therefore, to write a letter to John 
Brown, Chairman of the School Board, stating your views. 

In your letter; state clearly whether you are for or against the 12-month school year. 
Give reasons for your opinion. Be specific. Use examples from your own experience to 
support your answer. Sign your letter Tat Smith." 

Be as convincing as you can. Your letter will be scored on how persuasive it is. 



Scoring the Sample Test Item 
Simple Paper I: Score: 2 
Chairman Brown, 



Don't do it. Its a bad idea. For every person that likes the idea there will be 10 that don't. People 
will hate you for this lousy id^u Forget it!! 

Thats my opinon. 
Patrick Smith 

r 

Rationale: Though there is no clear statement of position, this writer's viewpoint is easy to in- 
fer; therefore, this paper is scored a 2. The paper has little else to commend it, however. The 
vocabulary is ineffective, the tone almost belligerent. The paper relies heavily on emotion; in- 
deed, support is virtually absent. 
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Sample Paper II: Store: 3 
Dear Chairman Brown, 

I like the idea of the 12-month school year and I will tell you why. 

First, I am too young to get a job during the summer (except for babysitting, which I do not 
like) so I might just as well be in school doing something. 

In the second place, having summer school will give work to all the teachers that are unem- 
ployed because they can't get regular jobs. At least they would have three months (about) of 
work and that is better than nothing. 

But the most important reason is that you will cut down on juvenal delinquancy. If kids are in 
school all the time they will not have as much of a chance to get in to trouble. Plus they will be 
learning more at the same time. About the only kids who will be against this is the ones who are 
lazy, and they will be against just about anything anyway. 

Thanks for your time. 
Pat Smith 

Rationale: This paper offers a clear statement of position backed by three distinct arguments. 
There is apparent awareness of audience; tone and wording are appropriate. The paper is, in 
fact, a borderline 4, and your rating depends on whether you consider the style rhetorically ef- 
fective. There are some awkward moments: "... three months (about) of work ..." and ".. . 
is the ones who are lazy ... " This writer has a good sense of logic and order, but needs a 
stronger vocabulary and greater skill in handling language. 

Despite the 3 rating, supporting evidence in this paper is not flawed or weak. Remember, no 
jiaper fits all scoring criteria. It is still necessary to exercise personal judgment. In this case, 
snappy style would have earned this otherwise satisfactory paper a 4. 



UNIT 6: PRIMARY TRAIT SCORING PRACTICE (PAPERS S-W) 

Unit 6 contains five papers, S-W. Score all five for the primary trait of persuasiveness, 
according to the guidelines given. After scoring, compare your scores with those suggested by 
the author. Remember, do not turn to the Rationale section until you have finished scoring 
papers S through W. 

Primary trait scoring generally takes 2-5 minutes per paper. Take your time. But if you 
are consistently taking longer than five minutes, you may be deliberating too much, or you may 
not be sufficiently familiar with the Scoring Guide. 



NOTE: If for any reason you think you may wish to reuse these materials, 
or share them with a colleague, we suggest you make copies of stu- 
dent responses before scoring. 
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PAPERS SCORE: ** 

Dear Chairman Brown, 

The 12-month school year might be a good idea for some but it is not for everyone. Some 
students need extra help. It is fine for them to go to school all year, but they get the help they 
need through summer school. Why impose this requirement on everyone? 

The 12-month school year is not needed. In many cases we spend days reviewing old material or 
just having discussions. There is not enough work to fill 12 months. Teachers would have to 
stretch the work out even farther and all of us will get bored. I feel this idea should be kept in 
mind for a future time and should not be acted on now. 

Thank you 

it Smith 




PAPER T SCORE: 
Dear Mr. Brown, 



The 12-month school year strikes me as heartless. I suppose youWl say the Board of Directors 
has the student s best interests as heart, but that is only your sidejof the picture. What do we 
have to lose? Recreation time, for one thing. We sit in the classroom all year and grow pale and 
aneaimc. We need to get out in the sun and fresh air in the summed. What about our social life' 
In school we justsee the same old people dav after day. This is not too stimulating. Our minds 
can get stale from this lack of activity. It is clear you will only defeet your own purpose with the 
12-month school year by making things worse than ever. I am sure if you and the other board 
oi directed discuss my ideas you will eventually come to see that I am correct. You mav con- 
sider this my protest to the 12-month school year. ,, 

Respectfully yours, , 

Pat Smith 

PAPER U SCORE: 

Dear Mr. Brown, 

There are two sides to your idea. Personally I like school and will not mind coming ever day. 
Not all my friends will feel the same way. 

To see the other side, you have not made clear your posishion on vacashions. To be spesific, do 
you plan to have any? If there are no vacashions 12 months could be a long time. This could ef- 
fect my vote. I can see we maybe do not need the whole 3 months every summer like we have 
had. But this change is too drastick. Think of a compromize so everyone can be happy. Then you 
will get my suport 

Thanks 

Sincerely -Pat Smith 
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PAPER V SCORE: * 

* Some of the most intelligent people in history have been self-taught. You do not need to attend 
school to be well educated. A wealth of information awaits us in books. For the man who can 
read all doors to knowledge and wisdom are opened. Most people learn to read in school, but 
even this is unnecessary. A person can learn from his parents or peers or even on his own if he 
has to. School is really a small, insignificant part of a person's educational experience. Reading, 
on the other hand, can bring knowledge, joy and satisfaction throughout a person's life. 

PAPER W SCORE: 

Dear Chairman Brown, 

My suggestion to you is give up on the 12-month school year. It r s an idea nobody but the Board 
will like. They do not h$ve to attend school so they do not understand how it is for us. 

Our hours are long and the work is very hard. By the time June comes along we are worn tp 
nubs. Not to mention the teachers who must suffer along with us. We need our summers off. 
We can't learn if we are too tired to see straight* 

But here's the real point; school costs money. Every time we open the doors, there go some 
more taxpayer's dollars. It costs money to buy heat, books, food, lights, and teacher's salaries. 
Where will it end. You cannot consider yourself a real leader of the community if you let things 
like this happen. 

There are more important things for the boardTO^Ual with. We need better parking facilities, 
better pay so we can hire some more teachers and more uptodate equipment for our gym and 
track team. These things won't wait. Therefore, do not waste our time and money with ideas 
nobody believes in. Take my advice; you will not be sorry. 

Sincerely, 

Your faithful student, \ 
Pat Smith 



PLEASE TURN THE PAGE FOR SUGGESTED SCORES AND RATIONALES. 



INIT 6: SUGGESTED SCORES AND RATIONALES (PAPERS S-W) 
PAPER 8 Suggested Score: : 3 



Rationale: S takes a dear stand (The 12-month school year is not needed"), but the arguments to 
support this stand are weak: i.e. ( it would be too boring, not everyone needs extra help, there's 
not enough schoolwork to fill a year. These opinions are not backed with evidence, nor are they 
presented effectively. The opening and closing lines are weak and ambivalent. Overall, the paper 
lacks zip and style. (Some raters will likely score this paper as a strong 2.) 

Agree , 1 " * "* 

_ Disagree 



Comments: 



PAPER T Suggested Score: 3 

Rationale: Here's a paper with a good strong opening that loses its punch about midway through 
and neve- regains momentum. The image of "pale, anemic" students deprived of "stimulating" 
social life adds a touch of humor, but isnt really a sound argument T, like S, finds school tedi- 
ous - fine, but you can't effectively structure an argument around one opinion. The last few 
sentences (beginning with "It is clear . . .") add virtually nothing. (Again, some raters may see 
this paper as a strong 2.) 



Agree 
Disagree 



Comments: 
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PAPER U Suggested Score: 1/2 



Rationale: Just what is U*s position? Who can say? This faltering little letter vacillates enough to 
make the reader dizzy. U says, in effect, if you really want the 12-month school year, ok/only 

aren't going' to be any vacations I 
y v!Sf e . my ^ ut ^.T* how) ' 80 mean wlule, come up with another idea and 
maybe 111 let you know how I feel. In short, U is polite, but vague. If you considered "Personally I 
Uke school and will not nund coming every day" a clear statement of position, you probably scored 
this paper 2; if not, you probably found little to elevate it above the ambiguous 1 level. 
NOTE: Uisa muddler, but not a totally incompetent writer. This is a poor 
piece of persuasion to be sure. The vocabulary is flat and the organization 
inept. Sentence structure isn't too bad, though, and there's a nice conversa- 
tional quality to the style. Suppose you were scoring the paper holistically 
W pdd your score be different? 

Agree 

Disagree * 



Comments: 
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PAPER V Suggested Score: 0 

Ratronale: Here's a thoughtful, intelligent little essay that has only one real problem: it doesn't 
address the question asked. Remember that zero is not an indicator of quality. V is an unusually 
competent ninth grade writer who simply didn't respond to the test item. In a large-scale assess- 
ment, the usual procedure is to give students like V a chance to retake the exam In the 
classroom, of course you can be more flexible. You may, for instance, even give students the 
chance (occasionally) to write off-topic if they choose. Remember, however, that you cannot pro- 
perly score such papers using guidelines developed for a specific item. 

Agree 

Disagree 

Comments: / 
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PAPER W Suggested Score: 4 

Paper W has much to commend it: forceful diction, good organization and paragraphs, 
reasonably good sentence structure, and perhaps most exciting of all, some real content to sup- 
port a strong stand. ("If s an idea nobody but the Board will like.") Wdoesn't just cite opinions; 
he/she gives reasons. This is refreshing. There are some flaws: a few cliches (e.g., "worn to nubs" 
- though tile effect is more humorous than annoying), misuse of the semLulon in the third 
paragraph (partially redeemed l>y correct use in the last sentence), faulty word choice ("It costs 
money to buy . . . teacher's [sic] salaries"). In general, though, Ws ability to ^± tenaciously to 
the centra] idea and to maintain interest through originality, style and sound thinking are un- 
matched by any other sample in this unit. Ws tone is occasionally a little impudent; if you were 
[Hit off by that, you probably rated this paper a 3. 

Agree 

Disagree 



Comments: 



SUMMARY 

Take a few moments to review your scores on papers S through W. If your scores did rot 
differ by more than one point in any case, you are in virtual agreement with the author, and are 
probably ready to begin constructing exercises for classroom use; 

If you disagreed significantly (i.e., a discrepancy oftwo or three points) with just one score, 
you can probably chalk that disagreement off to individuality. If, however, you dk^reed 
significantly with the suggested scores in more than one case, recheck any comments you noted 
in the Rationale section, then ask yourself the following questions: 

1. Did I allow myself time to become thoroughly familiar with the Scoring Guide before be- 
ginning to rate papers? 

2. In scoring, did I rea4 each paper through thoroughly rather than just skimming? 

3. Did I refer to the Guide frequently? 

4. Did I remember to score each paper strictly on persuasiveness without letting other 
factors (e.g., spelling) influence my judgments? 

NOTE: A Utter with lots of spelling errors probably isn't quite as persuasive as a letter 
wtthjlawkss spelling, other factors being reasonably equal. To the extent thai any factor, 
like organization, influences a primary trait like persuasiveness, that factor must 
somt w (directly or indirectly) be comulered in sc<yring. 



ERIC 



9' 5 



5. Did I let myself be overly influenced by the writer's viewpoint or vice versa (that is 
downgrading a paper because I disagreed with the writer)? 

6. Did I remember to place myself in the position of Chairman Brown, realising that the 
basic tenet of primary trait scoring holds that all writing is done in terms of an audience? 

REMEMBER: The main question to ask yourself here is, would this letter persuade me 
(as Chairman Brown) to adopt this student's position on the 12-month school year? If one 
were to translate the answer into a score, the results would look something like this- 
4 paper = Yes, definitely (would persuade me) <• 
3 paper = Y^s, probaHy 
2 paper * No, probably not 
1 paper = No, definitely not 

Additional practice 

*u ? SS W * 0ul . d ^ additional Practice, please turn to Unit 7, page 50 , and score papers AA 
through EE. As in the preceding units, suggested scores and rationales are provided. 
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UNIT 7: PRIMARY TRAIT SCORING PRACTICE (PAPERS AA-EE) 

Score the following five papers for the primary trait of persuasiveness, using the Scoring 
Guide provided for Unit 6. Enter all scores before checking the suggested scores and rationales 
that appear at the end of this unit. 

PAPER AA SCORE: 

Dear Mr. Brown, » 

As ideas go this one is not too bad. I figure if we go to school for 12 months we can finish up about 
three years sooner and be ready to stjart our lives when we are about 15 years old (14 for some). It 
will be a long hawl but worth it at tjhe end when we are free, FREE, FREE! 

Pat Smith 

PAPER BB SCORE: 

J Dear Mr. Brown ^ 

No to your plan It will never work Noone lakes your idea that much we do not lake shcool that 
much so it is a bad idea this is my opinon and my friends too 

Pat Smith 

PAPER DD SCORE: 

Dear Mr. Brown, 

Have you considered the consequences of your actions? A 12-month school year could create alot 
of unrest in students and teachers. Don't you tatiQh vacation? I am sure the Board of Directors 
does not meet the year around. We are only asking for those same privileges. We need your sup- 
port in order to maintain our mental health* If you create to much discontent in the schools people 
will cease to learn. The system will disintegrate around us. Teachers will resent students and we 
will be the losers* Its not too late. Reconsider your plan and give up the 12-month school year. In 
the end we will all benefit. 1 c 

, Sincerely vours 

Pat Smith 

PAPER EE SCORE: 

Dear Chairperson Brown: 

Basically I would say the answer to your suggestion is: no. Why do I say this? For two reasons. 
First of all, nobody (except a few kids who love school) want to go to school all year long. You will 
havea lot of disipfine problems when you try inforsing this rule. Second, when people do not like 
a rule they tend to go against it and pretty soon nobody obeys that rule anymore. You will be sit- 
ting in this great big school all by yourself if you try to make people go to school when they don't 
want to. I'm sure you were thinking of our welfair: but this idea is really the pits. 

Thanks anyway, 

Pat Smith f 

) 
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PLEASE TURN THE PAGE FOR SUGGESTED SCORES AND RATIONALES. 
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UNIT 7; SUGGESTED SCORES AND RATIONALES (PAPERS AA- EE) 
PAPER AA Suggested Score: 2 

Rationale: If s pretty clear that AA likes the idea of the 12-month school year, albeit for a ques- 
tionable reason. This letter is short and to the point. There's certainly no ambiguity, and there is a 
healthy dose of enthusiasm. However, AA makes no real ettort to persuade; he/she seems to ac- 
cept the 12-month year as inevitable. There's no logic, no strong content: in short, no argument - 
just a verbal nod. 

Agree . 

Disagree 

Comments: 



PAPER BB Suggested Score: 1 

Rationale: BB lacks even the limited persuasive power of AA, who had at least a reason for 
his/her opinion. BB's "logic" is roughly "I dislike this idea; therefore, it's a bad idea." Far from be- 
ing persuasive, this says practically nothing. Poor sentence structure and mechanics don't help 
matters. (Compare the readability of AA.) 

) Agree 

Disagree 

Comments: 



PAPER CC Suggested Score: 4 

Rationale: If one were to seek a textbook model response to Item 2, this would be it. The logic is 
sound, with two good reasons provided for opposing the 12-month school year: loss of time with 
family and loss of travel/work time. The language is bright and sophisticated, the tone 
businesslike yet sincere. It is hard to imagine any Chairman of the Board being unmoved by so 
cogent an argument as CC presents. 

Agree 

Disagree 

Comments: 



PAPER DD Suggested Score: 3 

RationalejDD offers only one main argument: the threat of unrest among students and 
teachers. That argument is presented clearly enough, but one senses that DD lacks the 
vocabulary and rhetorical skills to convey the full force of his/her emotion. The letter is redundant 
too. The notions of "unrest," "discontent," "people will cease to learn," "the system will 
disintegrate are little more than echoes of one another. Also, the tone here is a little accusatory 
* If you were put off by that, you may have rated DD a 2. 

Agree ' 

Disagree 

Comments: 



ERIC 



59 



PAPER EE Suggested Score: 2 

Rationale: EE certainly enjoys using colons. Unfortunately, neitherthis nor any other rhetorical 
device rescues this paper from mediocrity. EE promises the reader two arguments, but delivers 
only one in two packages. And even the one argument - that nobody will like the 12-month 
school year - is weak and certainly unoriginal. The letter disintegrates into nothing more than 
emotional fuzziness disguised as logic at the end (i.e., "You will be sitting ... all by yourself"). 
EE's reasoning is unsophisticated, unrealistic, and above all unpersuasive. 

Agree 

Disagree 

Comments: 



TO SEE HOW A RESPONSE TO ITEM 1 MIGHT BE SCORED USING THE PRIMARY 
TRAIT APPROACH (VERSUS HOLISTIC), PLEASE TURN THE PAGE. 
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COMPARING SCORING APPROACHES: HOLISTIC VS. PRIMARY TRAIT 

The following examples are included to illustrate how a response originally scored 
holistieally might be scored using the primary trait approach. The two responses shown here are 
taken from chapter II on holistic scoring, and the original holistic scores are shown here for com- 
parative purposes. 

In addition, these responses have been scored accordingto the Primary Trait Scoring Guide 
presented in Chapter IV. Readers are asked to keep in mind that these guidelines were developed 
for use with Item 2, and are used unmodified with Item 1 for illustrative purposes only. It is 
always best to develop scoring guidelines in conjunction with the item for which they are intend- 
ed. (For the reader's convenience, Item 1 is reprinted here.) 

ITEM 1 (Reprinted from page 8, ) 

Toby loved to hike in the woods. One day when he was hiking, he found a bird with an injured 
wing. Toby took the bird home, set his wing, and took care of hin. He named the bird Mike. Soon 
Mike got better, and he could have flown away, but Toby kept him in a small cage. Toby knew 
the bird was ready to be free, but he liked Mike so much he decided to keep him as a pet. 

Do you think Toby made the right decision? Write out your answer on this page. State clearly 
whether you think it was a good idea for Toby to keep Mike. Then explain your reasons as com- 
pletely as you can. Your response will be scored on how persuasive it is. 

PAPER G (Reprinted from page 15) 

I know from my own experience just how Toby feels. Once a bird flew in to our livingroom win- 
dow and wa« knocked unconsious. For a whole day it looked like the bird would die. When it came 
to it wouldn't eat and I had to feed it with a eye dropper. I kept it in a box in the garage. Then one 
day as I was taking it some thing to eat I discovered the box was empty! My beautiful pet bird had 
flown away never to return! At first I wished I had never seen this bird; then I would not have to 
feed so sad! 

But then I realized that at least I had helped the bird get well and that was better than having 
it for a pet. I would of kept the bird if I could, but it would have been wrong. So thinking about 
my own experience, Ibby should let the bird go free. 

Suggested Scores 

Holistic: 3 (See page 16 for rationale.) 
Primary Trait: 2 

Primary Trait Rationale: G takes a long time getting to the point, .and this delay damages 
persuasiveness. 3 has empathy, but strays from the track a great deal discussing personal 
experience. This paper could have easily earned a 3 had G presented the main idea first (i.e., 
"Ibby should let the bird go free"), then hacked it with a personal anecdote. Order is critical to 
persuasiveness. 
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Paper H (Reprinted from page IS) 

Tbby has done a fine thing in caring for this helpless bird. Without his assistance, the bird 
would probably have become the prey of a fox or hawk. (What kind of bird is Mike anyway — it 
doesn't say.) But then Tbby undid all the good he had done by deciding to make a pet of a bird 
that belongs free in the skies. It would have been better to let the bird die. Does that sound 
cruel? It isn't. Man should not interfear with Nature. At least Tbby should have known enough 
to release the bird once he had done all hecapld do to help it. So no, I cannot say Tbby has done 
the right thing. A person who knows enough to help a bird with a broken wing should also 
kno>v er ;>ugh to release it once the wing is strong again. 

Suggested Scores 

Holistic: 4 (See page 16 for rationale.) 
Primary Trait* 3/4 

Primary Trait Rationale: H has a strong tone of conviction that lends it great persuasiv- 
ness. One could argue that the main thesis — i.e., that man should not interfere with nature — 
would be better presented first. Also, the second to last sentence, intended as a clarification, 
really adds nothing but wordiness. H's contention that the bird would be better off dead than 
caged is somewhat startling, especially when so starkly expressed. H is forthright and strictly 
to the point: a no-nonsence kind of writer. As in G's case, slight reordering could have boosted 
persuasiveness, but overall, this is a fine effort. < 

NOTE: A similar comparison between holistic and analytical scaring ap- 
j pears at the end of Chapter III Users are encouraged to try this kind 
of amparison as a classroom exercise, to see how scores can 
sometimes differ with different scoring approaches, and to discuss 
the resultant meaning of those scores. 

THIS CONCLUDES THE CHAPTER ON PRIMARY TRAIT SCORING. THE FOLLOW- 
ING CHAPTER OFFERS SUGGESTIONS FOR CLASSROOM USE. 
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CHAPTER V: Suggestions for 

Classroom Use 



USING WRITING ASSESSMENT SCORING APPROACHES 
TO TEACH WRITING 

Through the preceding chapters you have had a chance to experiment with three 
methods of scoring student papers: holistic, analytical and primary trait. Once you are comfor- 
table in applying each of these methods, you can use what you know to enhance classroom in- 
struction. The following suggestions indicate a few ways in which this might be done. 

The emphasis here is on student involvement and the use of writing assessment to teach 
writing skills. Some teachers may also want to use one or all of these methods as a part of stu- 
dent evaluation. That's fine.* Remember, however, that students can derive great benefit from 
an opportunity to act as critics. By helping students learn to apply the principles of these three 
scoring methods, you can give them many of the skills they need to write well. 

A Word About Grading 

Educators are cautioned against using the methods described in this handbook par- 
• ticular y holistic and primary trait scoring - as a sole basis for assigning grades There are 
several reasons for this. 

First, depending on how classroom exercises are structured, students may not receive 
enough feedback to justify assignment of a letter grade. In a typical classroom discussion, cer- 
tain papers are bound to receive more thorough analysis than .others. Further, you as an in- 
structor may not gain sufficient information on which to base a grade. It is one thing to rank 
order students on the basis of, say, three of four holistic writing exercises. It is quite another 
thmg to use those same exercises to determine whether individual students have mastered five 
or ten basic skills.** 

v Second - and more important - by asking students to actually score papers, you are 
making them editors and critics. This is a role students often enjoy immensely, but more so if 
the threat of an impending grade is eliminated. You will have the greatest success if you create 
an open atmosphere in which students feel free to comment and criticize writing (theirs and 
others') knowing that no formal grade hangs in the balance. ^ 
Third, ratings on a four-point scale do not always convert readily to letter grades An 
analytical scoring system which incorporates five or six traits carefully delineated through ex- ' 
phot criteria may provide an adequate and appropriate basis for assigning grades. Indeed, of 
the three scoring approaches discussed in this handbook, analytical scoring provides the most 
specific mfonnation. But primary trait scoring, as we have seen, pinpoints only one facet of a 
students writing skill; therefore, a grade assigned on that basis cannot be reflective of a stu- 
dents overall ability. 



*ev3iS^ i0n m ^ U ° f *" Chaptel " deals s P ecificall y ^th the use of writing assessment in testing and 
"Analytical and primary trait scoring will, of course, provide more specific and detailed information than holistic. 
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An even greater hazard lies in attempting to translate holistic scores into letter grades. 
As noted earlier, holistic scoring is a normative system in which each rating reflects a paper's 
relative ranking within the group, quite apart from that paper's individual merits. If students 
(as a group) do exceptionally well on a given assignment, then even the 2s and Is may be quite 
readable papers. Conversely, if students (as a group) respond poorly to an assignment, even the 
4s may be mediocre. 

The three scoring approaches can all be used quite successfully as a means of discovering 
students' strengths and weaknesses. Used in this way, they are an excellent asset to an 
established program for student evaluation. In most cases, however, teachers will find these 
methods inappropriate as shortcuts to conventional grading practices. 
I 

Adapting the Author's Suggestions 

The suggestions presented here are designed to suit a range of teaching styles and stu- 
dent needs. Some will certainly be more appropriate for your stents' grade level and range of 
skills than others. * 

Keep in mind that these suggestions are only guidelines. You are encouraged to use your 
imagination in refining and adapting them as desirable. 

SUGGESTION 1: Scoring Papers 

Instruct students in any or all of the three scoring methods, then ask them to score the sample 
student responses provided in this handbook, using the range finders or scoring criteria provid- 
ed, as appropriate. 

Discussion: 

• Do students' scores agree with your own? 

• Do students' scores agree with those provided by the author? 

• If not, what priorities dp students reveal in evaluating writing? 



SUGGESTION 2: Let Students Write 

Give students a chance to respond to items 1 and 2. Score results using the range finders (for 
holistic)* or criteria (for analytical and primary trait) provided. 

Xeroxing all papers for students can be cumbersome. Consider using an overhead projector in 
scoring and discussing papers. 

SUGGESTION 3: Whose Paper Is It? 

Encourage students to "take the risk" of remaining anonymous by omitting names from papers. 
Criticism is more honest if no one knows whose paper is under discussion. Also try 

• Including a paper of your own for analysis. 

• Encouraging students to "disguise" telltale handwriting (distinctive handwriting is a 
sure clue to identity). 

• Allowing two comparable sections (e.g., two classes of freshmen) to score each other's 
papers. 



If you prefer, you can select range finders from among class papers. Keep in mind, though, that selecting range 
finders is time consuming because it requires reading all the papers from a small class (50 or under). For a more 
complete discussion of how to select range finders, see Appendix B. 



9 

ERLC 



64 



SUGGESTION 4: Keep It Lively 

Don't letstudents sit by as observers while you illuminate the flaws and merits of each paper. 
This should come from them. Ask students to be very explicit about what makes a 4 paper a 4, 
what makes a 3 better than a 2, and so on. 

• Ask students to comment on the benefits/disadvantages of having many "critics" as 
opposed to just one (i.e., the instructor). 

P 

SUGGESTION 5: Good Items vs. Bad 

Ask students to da a critique of Items 1 and 2. 

• Are directions clear? 

• Is the item stimulating? Is it fun to write about? 

• Is the item bias-free? That is, is there anyone for whom the item would be inap- 
propriate because of age, sex, or culture? If so, how could the item be amended to 
meet their needs? 



Compare the following sample items o the originals. 
Item 1 

One day a boy found an injured bird. He took the bird home and made a pet out of him. 
How do you feel about this? 

Item 2 

What if someone said you had to have" school all year around? Write a letter stating your 
views about this. 

SUGGESTION 6: Item Writing 

Give students a chance to write original items* using whatever guidelines come out of the 
discussion from Suggestion 5. As a minimum, items should be - 

1. Clear and complete, with unambiguous directions,, so that respondents understand 
all that will be expected of them, and are aware of how their responses will be judged. 

2. Bias-free, so that all students to whom the item is directed will have an equal chance 
to respond well. 

3. Stimulating, so that respondents are motivated to do their best. 



SUGGESTION 7: Item Testing 

Select one or two of the best items from Suggestion 6 for students to write on. Score results by 
whatever method seems most appropriate. In scoring, be sure to note the relationship between 
the quality of the item and the quality of the response. 
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SUGGESTION 8: Rating Outside Writing 

A good writer is a skillful critic. Once students are familiar with the three- scoring methods, 
have them try their evaluative skills on a piece of outside writing: 

• Newspaper/magazine clipping. 

• Student writing from another class 

• Ph.D. thesis 

• Advertising copy 

• Song lyrics 

• Business letter 

• Portion of novel or poem 

• Sample of your own writing 

• Movie script 

• Script from play 

• Papers from other classes (science, history, psych, sociology) 



SUGGESTION 9: Getting Into Real Writing 

A common complaint - among students and teachers alike - is that writing assignments 
aren't ?real.* Why not give students a chance to do some real writing - ^say a letter to the local 
newspaper editor? Score responses for format, tone, content, whatever - mail the best 
responses in. Getting a letter printed in the paper can provide greater reinforcement than any 
letter grade. 



SUGGESTION 10: More Real Writing / 

As a followup to Suggestion 9, try these exercises in "real" writing: 

• Find samples of actual college/scholarship applications. Many require a short essay. 
Responding to these - and getting reactions from peers - can be very worthwhile 
practice. 

• Have students draft letters of application in response to actual job advertisements. 
Let classmates take the role of "employers" in evaluating letters to see who "gets the 
job." ^ 

• Encourage students to write and mail in letters criticizing or commending a TV show 
in terms of content, performance, story level. Ask students to share any responses 
that are received. 

Students themselves may have additional suggestions on "real" writing. 



SUGGESTION 11: The Business Letter 

Mastering proper business letter format is a stumbling block for lots of students. Primary trait 
analysis provides a natural vehicle for teaching this form. But be careful; don't go too fast. 
Score letters just on format first. Later, if you wish, take an analytical approach by adding 
more factors: content, tone, organization, style. 
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SUGGESTION 12: Discuss the Factors 

The materials in this handbook deal with three factors - ideas, wording and organization - 
that might be scored analytically. 

• Ask students to identify others. 

Important:, It will be necessary to develop new scoring criteria for any new traits that 
are to be scored. You will find this process (as with range finders) simpler and faster if 
you develop the criteria yourself, though you may want to involve students in discussing 
and refining criteria. If you choose to involve students in the development of criteria 
prepare to spend a minimum of 2-3 class hours per trait. 

• Score some writing samples, using the factors identified. 

• Discussion: Are some factors more important to good writing than others? 



SUGGESTION 13: What About Mechanics? 

Even for experienced raters, the development and application of scoring criteria for rating 
mechanics is difficult because there are countless factors. to be considered. (It can take pages of 
criteria just to cover punctuation!) Therefore, it is NOT RECOMMENDED that you attempt to 
score mechanics usmg either analytical or primary trait scoring methods * 



SUGGESTION 14: Finding Other Primary Traits 

Ask students to identify the primary (that is, most important) trait in 

• A letter to the editor ' \ •-• » • A chemistry textbook 

• A biography • a movie script 

• A resume - • a radio ad 

• A letter of application • A news broadcast (script) 

• A recipe 

Provide examples and score them. 

Or - turn the exercise around, and ask students to name a piece of writing in which the 
following would be the primary trait: 

• Style • ideas 

• Organization • Originality 

• Tone • i ns ight 

• Wording • Accuracy 

• Clarity • Any others . . . ? 
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SUGGESTION 15: What Is a "Primary" Trait? 

So-called "primary" traits are sometimes umbrella terms that encompass component traits. 
Forinstance, organization may be part logic, part word order, part completeness, and so 
on. 

Dissecting such umbrella terms can help students understand what concepts like "organiza- 
tion" are all about, and how such qualities can be achieved in writing. 

Try dissecting - 

• Persuasiveness 

• Originality 

• Tone 

• Syntax 

• Diction 

Such words, part of the language arts terminology, are often meaningful to students only 
after they are broken into component parts. 

SUGGESTION 16: Focusing on Specifics 

Holistic, analytical and primary trait scoring methods complement one another when used as 
part of instruction. 

For instance, suppose that through holistic or analytical scoring practice, you find that many 
students are weak in one area: say, organization. To build skills, concentrate on the factor, scor- 
ing for ofganization as a primary trait. Be sure to use exercises in which organization - or 
whatever trait you wish students to focus on - is important. 



Examples (for organization): 

• Set of directions 

• Logical argument 

• Chronological account 

Of course, it isn't necessary for everyone to do the same thing all the time. Let students iden- 
tify their own weaknesses. Then score one batch of papers for organization, one for wording, 
and do on. 



SUGGESTION 17: If Students Find Analytical Too Hard 

For many students - even for teachers - analytical scoring can be overwhelming at first. It 
is demanding because it requires total concentration and rigorous application of explicit 
criteria. 

To simplify the process a little try this: 

1. Remember to have students score one factor at a time. That is, if ten papers are to be 
scored, review all ten for organization, then for wording, and so on. 
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2. Teach one fector at a time and do not go on until students feel ready. Gradually add 
others until students feel coiK 'able rating a single paper for two three or more 
*£! <f™ st «ff * will f ^fficult to g go bey^nd^rNolSSow JZy 

sS^usr y ored ' howevei ^ r to ^ *»» ° ne at a ti - - « 

SUGGESTION 18: Helping the "Straight A" Student 

For some students, the prospect of failure is intimidating - almost incapacitating. Help 
students understand that everyone, including the best students will occasionally receive low 
ratings. 

Students with consistently high expectations may cope better with what they perceive as 
negative feedback if ^ 

• Ratings are regarded only as indicators of relative achievement (e.g., "This paper 
has merit but does not do as much with organization as this one . . . ") 

• Ratings are not translated directly into letter grades. 

• Instructors are open enough to submit their own writing for analysis - even when it 
means accepting an occasional low rating gracefully! 

SIT^ESTION 19: Comparing Approaches 

Sjjudents who have had a chance to try all three scoring approaches (i.e., holistic, analytical 
antrpnmary trait) are in a good position to make some comparisons. Here are some brief sum- 
manes to launch the discussions: 

Primary trait scoring presumes that writing is done in terms of an audience, and that suc- 
cessful writing will have the desired effect upon that audience thahks to its excellence with 
regard to a specific factor: namely, the primary trait. 

Analytical scoring presumes that there are a handful of universal traits important to virtual- 
ly any piece of writing in any context, and that thorough analysis demands attention to ea<-h 
of these factors. 

Holistic scoring presumes that writing possesses some elusive quality that transcends factor- 
by-factor analysis: in short, that the whole is always more than the sum of its parts 
Therefore, any piece of writing is best judged for its overall effect. 
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SUGGESTION 20: Getting Fresh Ideas 

Consider forming a discussion group with colleagues to 

• Evaluate student papers 

• Talk over classroom adaptations 

• Write and review new items 

• Develop scoring criteria 

/ 



NOTE TO THE READER 

We \wpe this list of suggestions provides a good beginning for you in adapting scoring methods 
for classroom use. As you implement these suggestions, new ideas will undoubtedly occur to 
you. In addition, you will likely find ways to modify these original suggestions to make them 
work better for you. 

When you find an idea that works well, won't you let us hear from you? With your help, we can 
keep our list of classroom suggestions as up-to-date and complete as possible. We hope to issue 
expanded and refined versions of this list in the future. 
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USING WRITING TESTS AND TEST RESULTS IN THE CLASSROOM 

This discussion is directed toward the educator who wishes to use writing assessment in 
testing or as a part of student evaluation. Please keep in mind that writing items used for 
evaluative purposes - especially when performance affects a student's progress, grade or 
status - should be carefully developed, reviewed and pilot tested to ensure appropriateness 
quality and freedom from bias. 

TESTING AND DECISION MAKING 

Tests can serve various purposes in the classroom. One important purpose is to facilitate 
educational decision making. A test should not be administered, therefore, until any decisions 
that rest on the results of that test have been clearly articulated. 

To illustrate how writing tests can help teachers make educational decisions, each of 
eight contexts is described here in terms of the decision to be made and the type of information 
needed to make that decision. These descriptions are followed by a chart showing which of the 
three sconng approaches described in this handbook can provide the best information for use in 
each context. 

USING TESTS TO MANAGE INSTRUCTION 

t Diagnosis. Teachers often use tests and other performance indicators to track each stu- 
dent s level of development, diagnosing skills already mastered and identifying skills yet to be 
learned. Direct writing assessment via writing samples can provide much diagnostic informa- 
tion thus simplifying student grouping or instructional scheduling. Diagnostic information 
gathered over time using writing samples can provide a partial basis for grading or com- 
municating progress to parents. 

Placement. Educators must determine a student's level of development in order to place 
that student at the instructional level best suited to his/her skills. Writing skill tests, along with 
grades and previous courses completed, can be used to rank order and properly place students. 

Guidance and Counseling. In deciding their future educational or vocational activities 
students need to know how their writing skill compares to that of other students with whom 
they could compete. Writing test results can help students, their parents and their guidance 
counselors answer students' typical questions: Should I pursue advanced training in a 
postsecondary educational program in which writing is a key element? In which school or job 
am I most likely to be successful? Though test scores should never serve as the sole basis for 
answering such questions, they can play a valuable role. 

USING TESTS TO SELECT STUDENTS 

Admissions. In times of limited resources, it is not uncommon to have more candidates 
for a program than program openings. When this happens, educators must decide who will be 
admitted. Selection decisions most affect those at either end of the skill continuum. That is, 
more able students are selected for inclusion in advanced writing programs, while less able 
student are selected for remedial writing programs. Writing tests can be used to facilitate 
such selection. 

Certification. Tests can be designed to verify and document a student's knowledge or 
mastery of required skills. Teachers might use writing tests to certifv masterv.of hemnninp- 
writing skills for purposes of grading or promotion. District and state administrators might use 
minimum writing competency tests as criteria for high school graduation. 
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USING TESTS TO EVALUATE PROGRAMS 

Formative Evaluation. In formative program evaluation, teachers attempt to determine 
which instructional activities are functioning as intended and which need refinement. By 
testing students at periodic intervals in the course of a writing program, teachers can deter- 
mine which skills have been mastered and adjust instruction accordingly. Further teoting to 
check the effectiveness of any program modifications may be desirable too. 

Summative Evaluation. Summative evaluation reveals a program's overall merit, and 
suggests whether that program should be continued or terminated. Tests designed to assess 
students' performance on final learning outcomes are an important part of such an evaluation. 
As with formative evaluation, multiple test administrations are common. Tests may be given 
immediately prior to and following instruction to document program impact. 

Survey Assessment. Survey assessment refers to the collection of group achievement 
data to determine general educational development in any area, including writing. Information 
may be gathered by administering a writing test to a carefully selected random sample of 
students in the target population. Survey assessment is often cyclical, so that test reviewers 
can examine trends in writing skill over time. This information is of particular value to the 
public, as well as to building-, district- or state-level administrators who allocate resources by 
need. 



SELECTING SCORING PROCEDURES 

Selecting a scoring procedure is an important part of assessment planning since this deci- 
sion is closely related to the purpo&j for assessment. Though it is possible to conceptualize in- 
stances within each of the eight educational assessment contexts in which any given scoring ap- 
proach could be employed, the "best" approach will be that which provides the kind of informa- 
tion needed, at the correct level of specificity and in the most cost efficient manner. 

To illustrate, diagnosis of individual student strengths and weaknesses demands the 
level of specificity provided through analytical or primary trait scoring. Placement and 
guidance, on the other hand, may only require holistic ratings becausje the goal here is simply 
to rank students on a continuum of writing skill. Because holistic scoring is faster (and 
therefore more economical) than other hand scoring methods, it would be the most cost effec- 
tive approach in this context. 

Selection for admission may only require a holistic ranking of students, while certifica- 
tion may be done through holistic ratings or analytical or primary trait scoring, depending on 
the specificity of the competencies to be certified. 

Holistic scoring procedures are well suited to the relatively broad, unfocused nature of 
large-scale survey assessment However, analytical scoring may serve as well if the desire for 
individual data justifies the additional time and expense. 

Scoring procedures for formative evaluation depend on the specificity of the enabling 
and terminal objectives that guide instruction. If general writing proficiency is the program 
goal, analytical scoring may be selected. However, if instruction focuses on situation-specific 
rhetorical skills, primary trait scoring may be must appropriate. In r .ost instances, formative 
evaluation demands more specific information that holistic scoring provides. 

With su^uitive evaluation, holistic ^ss^nipnt may provide sufficient data to judge 
program viability. However, if stated program goals subdivide writing skill inlo component 
parts, analytical scoring may be appropriate. Since instructional programs in writing deal with 
writing of many types, for many purposes, primary trait scoring will have limited value in this 
context. 

The relationship between decision and scoring procedure is summarized in the following 

table. 
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WRITING ASSESSMENT PROCEDURES 
AS A FUNCTION OF ASSESSMENT CONTEXT 



Assessment Procedure 



Primary 
Holistic Analytical Trait 

X* X 
X X 



Decision to 
Context be Made 

Diagnosis Determine skills mastered 

Placement Match student to 
level of instruction 

Guidance Rank students for 

educational planning 

Admission Rank examinees for selec- 
tion into programs 

Certification Determine mastery of 
specific competencies 

Formative Determine parts of instruc- 

E valuation tional program in need of 
revision 

Summative Determine program viability X X 
Evaluation 

Survey Check status of student X 

Assessment educational development 

NOTE TO THE READER 

We hope this handbook has been useful to you, and that you feel confident now in introducing 
scoring techniques into classroom instruction. As noted in the Preface, this handbook can be 
used alone or in conjunction with a workshop on scoring methods (see Appendix F, page 93 for 
sample agendas). If you are interested in iruch a workshop, please contact: 

Richard Stiggins 
C APT Coordinator 



X 

X X 
X X 



Northwest Regional Educational Laboratory 
300 S.W. Sixth Avenue 
Portland, Oregon 97204 
(503) 248-6800 

To check what you have learned, we invite you to take the posttest on page 69 , and com- 
pare your score with that you obtained on the pretest, page 1 . Answers are provided in Ap- 
pendix G, page 95 . 
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POSTTEST 
Test of Writing Assessment Knowledge 

NAME: SCORES: 

BEFORE TRAINING 

AFTER TRAINING 



1. List the major steps in the writing assessment process. 

2. List three different approaches to scoring samples of student writing. 

3. Name the scoring approach that relies on an overall impression of the quality of a given 
piece of writing. 

Name the scoring approach based on the premise that all writing is done to communicate 
with a specific audience. 

5. Which scoring approach takes the most time to use? 

6. Which scoring approach takes the least time to use? 

7. Which scoring approaches are based on prespecified performance criteria? 

8. If you wanted to evaluate specifically how well students follow proper format in writing a 
business letter, which scoring approach would you use? 

9. Which scoring approach provides the most information about specific strengths and 
weaknesses in student writing? 

10. What is a "range finder" paper? 

11. Which of the scoring approaches is probably most subject to a "halo effect"? 

12. Probably the best way to use writing assessment in the classroom is to show students how 
you are going to score their papers. 

True 

False 

COMPARE YOUR SCORES ON THE PRETEST AND POSTTEST. ANSWERS ARE 
PROVIDED IN APPENDIX G. 
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APPENDIX A 

Answers to Commonly Asked 
Questions on Scoring 

Achieving agreement with other raters is largely a matter of practice. In addition, 
however, it requires understanding of the way in which scores are assigned. It may be helpful, 
therefore, to claiify some of the most commonly raised issues regarding scoring. 

1. In holistic scoring, must a paper be free of all mechanical errors in order to 
receive a 47 

By no means Mechanics is an important consideration, but no more important than 
other factors like style, syntax, word choice and organization. A 4 paper need not be 
flawless in any of these areas, though in practice most 4s will be excellent in most 
respects. The relative merit of a 4 paper in holistic scoring depends on the quality of the 
sample as a whole. 

2. What if a paper is well written but does not address the topic? 

In large-scale assessment, a special category is created for papers that are illegible, 
btank, written m a language other than English, or that address a totally different topic 
This category may be assigned a number off the scoring scale (e.g., 7 for a 4-point scale) 
or may be designated the "zero" (i.e., "no score") category. 

In state exams, students who receive a zero (no score) are usually given a chance to 
repeat the exercise. This is one option yOu can use in class too. Remember that you can- 
not score a paper on one topic using range finders or scoring criteria developed for 
another topic. 

3. Shouldn't an extremely poor paper receive a zero? 

Definitely not. No^paper, regardless of how poorly written, is ever qualitatively below 
the 1 level. "Zero," if used at all, denotes a special category as described in response to 
question 2. 

4. Are longer papers generally better papers? 

Raters must take care not to be unduly influenced by length. Though it often happens in 
practice that skillful, articulate writers do write ihe most, length in itself is not in- 
dicative of quality. A concise, well conceived response may well merit a 4 rating. On the 
other hand, a wordy paper, weak in content, may deserve only a 1 or 2. 

5. Is handwriting a reliable indicator of a paper's quality? 

Generally speaking, no. Handwriting, like length, is a tempting factor on which to base 
a hasty judgment - especially when raters are tired or bored. However, handwriting is 
not a reliable indicator of overall quality. Of course, if s O.K. to consider handwriting, 
along with other factors, provided this decision is made in advance and respondents 
know what's expected of them. 

In analytical or primary trait scoring systems, handwriting (like any factor) must be 
noted in the scoring criteria if it is to receive consideration. Similarly, in holistic scoring, 



9 

ERIC 



75 



71 



rangedinders should provide some definite indication of the extent to which hand- 
writing is to influence scores. 



See Appendix C for examples of handwritten student responses. 

61 Is a 4 paper always an excellent paper? 

Usually - but not necessarily. In analytical scoring, 4s are generally of high quality 
because they must meet prespecified criteria. Of course, the quality and completeness 'of 
the criteria themselves determine in large part how meaningful scores will actually be. 

In primary trait scoring, a paper must also meet prespecified criteria to be judged a 4. 
However, if only one trait is being evaluated, it is quite possible that a paper which 
earns a 4 in that category is far from excellent in other respects. 

Holistic scoring is more complex. Remember, the range finders on which holistic scores 
are based are generally drawn from the sample of responses. (Other range finders may 
be used only if they reflect the general ability level of the respondents.) Theoretically it 
is possible (though admittedly unlikely) that out of say, 100 persons, no one would pro- 
vide what could objectively be termed an excellent response. This would most likely hap- 
pen if the test item were ambiguous or otherwise flawed. In that case, a 4 paper, while 
one of the better papers in the sample, might not be an excellent paper in its own right. 
Given a sufficiently large sample of writers with varying abilities, this situation is cer- 
tainly a remote possibility. However, it could occur with a very small sample - say, 30 
students - particularly in a special grouping where students were all, more or less, of 
similar ability. 

Educators planning to adapt holistic scoring for classroom use can minimize the prob- 
lem by ensuring that items are of high quality, and appropriate to students' levels of in- 
terest and ability. 

7. What are the qualities of a good writing assessment item? 

If you develop your own items for use in class, it's important that you attend carefully to 
the quality of those items, for inappropriate items cannot be properly scored. 

First, be sure items are complete. In other words, provide all information that a stu- 
dent needs to write a good response: e.g., names, dates, addresses for letters, key 
words and descriptions. In all cases, the type of audience to be addressed and purpose 
for communication should be specified. 

Second, write clear directions. If length is to be a factor, say so. If you want the stu- 
dent to take a definite stand on an issue, make that clear. Will handwriting count? Let 
the student know. Be sure with letter items to provide a name and/or title to whom the 
student should write. Letting students know in advance how papers will be scored 
(e.g., for persuasiveness) is not cheating. It helps students focus their thinking and 
makes the experience more instructional. In general, the specificity of instructions 
should increase along with the specificity of skills to be assessed. In other words, a 
survey of overall class skills demands less specificity than diagnosis of each student's 
strengths in, say, structuring a convincing argument. 

Third, review all items and revise as necessary to eliminate potential bias. Before 
using any item, ask yourself whether there are any groups to whom the item could be 
offensive or for whom it might prove unduly difficult. In short, items should be geared 
to respondents age and interest level, sex, cultural background and experience. Sexual 
bias can be particularly difficult to spot. An item which is of greater interest to boys 
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than girls (or vice versa), or which deals with a subject that one group is likely to know 
more about, is sexually biased. If you are unsure, students themselves can sometimes 
offer guidance here. 

Finally, make items interesting. The only thing worse than responding to a boring item 
is having to score the mundane responses youH get. Stimulating items inspire students 
to do their best. 

(See Appendix D for a list of potential items and sources for obtaining additional items.) 
& In holistic scoring, will the number of scores at each level follow a hell curve? 
Given a sufficiently large sample and range of abilities, yes. As a general rule, expect 
large numbers of 2s and 3s, relatively fewer Is and 4s. 

If you are consistently scoring a large percentage of papers 4 (or 1), one of several 
things is probably happening: 

• Your level 4 range finder paper is not truly representative; probably, it's just a 
strong 3. Or (in the case of all Is), your 2, 3 and 4 range finders are not represen- 
tative. More likely, you have taken the three single best papers from the class and 
mistakenly labeled them representative of a larger group of papers. Ideally, the 
level 4 range finder should represent a small group of papers at the top of the scoring 
scale, the level 1 ranger finder, a small group of papers at the lower end of the 
scoring scale. 

OR- 

• Your students are so close to one another in ability (at least on a given item) that 
you are not getting a real range of responses. 

OR- 

• Your item is not challenging the more capable students; thus you are getting "ar- 
tificial" 4s that really exemplify mediocre effort. Or (in the case of all Is), the item is 
so difficult that even the best students cannot respond well. 

The bell curve should be a general expectation, not a goal to strive for, however. Score 
as you see fit, without aiming for a specific number of 4s, 3s and so on. You won't get a 
perfect bell every time. 

9. Can primary trait analysis cover more than one trait? 

Yes. Usually, however, one trait is deemed the most important to a given piece of 
writing and thus is termed the primary trait. Additional traits to be scored are termed 
secondary and tertiary traits. 

If a wide range of traits is to be covered, you might ask yourself whether you are not, 
in fact, doing analytical scoring. 

10. Is it all right to split scores (e.g., 213 or 314)? 

Yes. Occasionally, papers are too close to call. Remember that in writing assessment, 
scores actually represent a range of achievement along a continuum. Thus, a high 2 
may be very close to a low 3. 

In assigning a split score, you are in effect, simply expanding the rating scale - you 
are giving yourself elbow room. A 4-point scale, after all, does not allow for extremely 
tine discriminations. Some inters arc much happier with an 8- nr 9-nnint scale, but 
such a scale is harder to master, expecially for student raters. 
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In short, feel free to assign split scores when a decision is really difficult - but do not 
use it as a crutch to avoid decisions. 

11. In scoring papers, why is agreement with other raters necessary or desirable? 

Agreement is merely a measure of consistency. If you agree with other raters regar- 
ding what score a paper should receive, you have some assurance that your ratings are 
based on sound criteria or guidelines (e.g., range finder papers). This firm basis makes 
ratings more meaningful, and more defensible as well. (Certainly students like to think 
that ratings are not the result of an arbitrary decision.) 

The notion of agreement allows for some flexibility in assigning scores, however; no two 
raters will ever agree totally - nor should they. Writing is not an exact science. 
Disagreements are important because they often arise when one rater notices 
something about a paper that another has overlooked. In large-scale assessment, 
disagreements are generally resolved through a third reading. In the classroom, 
disagreements can be a fine basis for discussion and learning. 

Remember that consistency is particularly desirable if scores are to be used in determin- 
ing student placement, promotion or grading. 

12. Can persons other than experienced language aria personnel be trained to score 
student papers? 

In many cases, yes. In holistic scoring, perhaps the most important qualification is that 
one be an alert, sensitive reader. It is not necessary to be at home with terms like "poor 
diction or "faulty syntax" to recognize muddled writing. Further, raters have range 
finders to assist them in scoring papers. 

The ease with which non-language arts personnel can learn analytical or primary trait 
scoring depends largely on the nature of the criteria used. If language arts terminology 
is an integral part of those criteria, the task may prove difficult. There is nothing par- 
ticularly formidable about the process itself, however. 

At the same time, there is little question that experience in reading and rating student 
papers makes one proficient at the job. Persons who have not taught writing and who 
are totally unfamiliar with typical student writing generally have more difficulty adapt- 
ing to a particular scoring approach than do language arts educators. 

13. Can range finders be used with analytical or primary trait scoring? 

Theoretically, yes. In practice, however, the use of range finders in analytical and 
primary trait scoring is cumbersome - for two reasons. 

First, raters are already dealing with a set of criteria. Having to internalize and refer to 
both criteria and model papers is time consuming and sometimes downright annoying. 

Second, having to identify representative papers for five or six analytical traits, at four 
distinct levels of performance per trait, would be a difficult, if not utterly mind boggling 
task. Experienced scorers are too well aware how few papers actually fit neatly into any 
prespecified category, regardless of how carefully developed criteria may be. (And who 
would really want 24 range finders to refer to?) 
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/ With primary trait scoring, where only one trait is under consideration, the use of range 

/ finders is conceivable if the idea really appeals to v ou. It is doubtful, however, that you 

/ will gain much in the way of simplifying scoring. Probably the best advice is this: try it if 

/ you must, and see if it works for you. 

14. Why must new criteria be developed for use with each item (exercise)? 
V Ideally, a teacher or other test developer writes an item to measure a particular skill or 

\ set of skills. The criteria for evaluating responses must be explicit enough to reflect 

y whether the skill(s) the test developer/teacher wanted to measure have been 
demonstrated. Only highly specific criteria, developed in conjunction with a given item, 
will do the job. To see why, consider this example. 

Suppose one wanted to know whether students could organize information well, and to 
measure this skill administered the following test item: 

SAMPLE ITEM 

Write instructions for baking an angle food cake. Be sure to include 
all steps and list them in the proper order. Begin with assembling 
ingredients and end with serving the cake. 

Now suppose that raters were asked to score responses using the following general 
criteria: 

4 = well organized 

3 = quite well organized 

2 = fairly well organized 

1 = poorly organized 

These criteria, which seem to delineate four levels of performance, are in reality mean- 
ingless. Words like "well," "fairly," and "poorly" are highly subjective and open to inter- 
pretation. Using such criteria, raters can be expected to achieve little agreement in 
assigning scores. Moreover, scores will have almost no meaning because the criteria - 
written to be generalizable to any item measuring organizational skills - are too vague 
to offer specific information about performance. 

Compare the following criteria, written explicitly for use with this item: 

4 = Lists all necessary steps in the proper order. 

3 = Lists all necessary steps with only minor errors in order. 

2 = Omits one or two necessary steps and/or has significant (i.e., the cake 

would not turn out) errors in order. 
1 = Omits more than two necessary steps and/or has such problems with order 
that one could not follow the directions. 

Raters using these criteria would have a far easier time assigning scores, and would be 
% more likely to agree on those scores. The criteria reflect what is asked for in the item; 
therefore, they are appropriate and fair. 

At the same time, these criteria would clearly not be suitable for use with another item, 
even if that item were intended to measure skill in organization. Describing the events 
that led up to the Civil War, for instance, is quite different stuff from outlining a recipe 
for angle food, and well developed criteria must respect such differences. In short, good 
criteria are thorough and explicit, and complement the item for which they are intended. 
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APPENDIX B 
Selection of Range Finders 

The Importance of Knowing the Sample 

A range finder paper must be so typical of all papers (within the sample) at a given level 
that virtually all raters would agree on the score assigned to that paper. Only then can a range 
finder serve as a model upon which to base the assignment of other scores. Finding a "typical" 
paper takes some doing. 

Suppose, for example, that a given sample comprises 40 papers. Joe Rater reads the first 
paper and finding it to be quite good, assigns it a 4. However, to Joe's surprise, the next half 
dozen papers are either equal in quality or superior to that first paper. The seventh paper is far 
superior. By the time Joe has read every paper in the sample, it becomes clear that at least half 
the papers are as good as or better than the first, about half the papers are not so good as the 
first. To properly show this relationship, Joe should have assigned that first paper a 3. But the 
point is, there was no way he could have known that in advance. Only by reading through the 
entire sample could Joe see how one paper compared qualitatively to all others. 

Holistic scoring is a normative system. In holistic scoring, scores are not based upon 
preset criteria; they have meaning only in relation to other scores. In other words, a 4 is a 4 
not so much by virtue of its being a fine paper in its own right, but by virtue of its being one of 
the best papers in a given sample. As Joe Rater found out, one can only see where a given paper 
fits in a sample by being thoroughly familiar with that sample. If there are 1000 papers to read, 
it is probably sufficient to read a random sample of 10 percent - or about 100 papers - to gain 
that familiarity. With a very small sample though - say the usual 25-40 papers from one class 
- it's highly advisable to read the whole sample (every paper) before deciding what is or isn't 
typical. Taking shortcuts can skew the rating scale and render scores meaningless. 

The Fastest Method 

The easiest method of selecting range finders for classroom use is to pick them yourself 
from the sample. Simply skim through the 30 or 40 class papers you wish to score, extract the 
four "typical'' papers (i.e.,\ one for each of the four score levels) you wish to have as range 
finders, and xerox or transcribe to overheads for class use. 

An Alternative* 

You can also write your own range finders. Do not attempt to do this until you have read 
through the sample, however; it's important that you be familiar with the range of responses so 
that your range finders will be representative. The chief advantage in this method is that it 
allows you to give students specific examples of what to do (4s and 3s) and what not to do (2s 
and Is). The particular points you wish to illustrate may or may not occur in the actual sample. 

A Longer Method: Class Involvement 

If you want to involve your class in the range finder selection process, prepare to spend 
considerably more time. You will, of course, have a fine opportunity to teach students, through 
comparison, what makes good writ'ng. Here's how it might be done with a class of 24 



•The alternatives presented here are intended for informal classroom assessment. Large-scale, formal assessment 
demands stringent procedures for the selection of range finders. 
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1. Divide the class into four small groups of six. 

2. Provide each group with six papers, randomly selected (provided papers are anony- 
mous, a student's own paper can be included in the group. 

3. Code all papers so that one group will not know what scores another group has 
assigned (e.g., Group 1 reads papers A-F, Group 2 papers G-I, and so on). 

4. Ask each group to select a 4, 3, 2 and 1 paper. (This should take no more than 15-20 
minutes.) 

5. Ask one student from each group to record scores using the code to identify papers. 

6. As soon as all groups have finished, have groups exchange papers and score again. 

7. Whan all five groups have read all initial "range finders,"* check for agreement on 
scores by a show of hands. (Only recorders need participate since they have marked 
down all scores for each group.) 

8. Select papers with universal agreement as range finders. 

NOTE: It is not necessary that an original "set" of range finders be 
selected. That is, it's ok to use one group's original 4, another's 3, 
and so on. 

9. Score remaining papers using the range finders selected. 

Don't be surprised if you have to spend two or three class periods or more selecting range 
finders. If this seems like too much time to invest, you may wish to select range finders yourself 
in advance. Remember though that in the process of selecting range finders, students are in 
fact rating papers and discussing what makes one superior to another: this is the heart of what 
the holistic scoring experience can provide. 



'You can take a shortcut by tallying scores after three readings. If agreement is good, chances are that two more 
readings won't change things significantly, so you can feel confident using the chosen papers as range finders. 



APPENDIX C 
Examples of Handwritten Responses 

The following examples illustrate how the quality of handwriting may confound scoring 
results. As these examples show, the best papers are not necessarily those with the neatest 
penmanship. 

HOLISTIC SCORING 
PAPER A SCORE: 2 
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HOLISTIC SCORING 
PAPER D SCORE: 4 




CLKjyy^ .Jul 
~tkls JyoA^j Jul Ul c(h jxiixHAlO, 
ffC (MoJjMU- Cri cy&- -httfitL. 
Mtt til, JliAXL 
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ANALYTICAL SCORING 

PAPER K SCORE Ideas: 4_ 

\ Wording: 4_ 

Organization: 4_ 

Total: 12 



vpooA, . ^ Irir *4 a^utusfaa&^c /MfxSf v^rtdt* ^ 

luh^UK. 7r»te bz^u, u< (lt^i^^^ 
do Jbpwr^ -Ml /3-#?**fM A^^-M 
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ANALYTICAL SCORING 

PAPER L SCORE Ideas: 

Wording; 



Organization: 1_ 

Total: 5 
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PRIMARY TRAIT SCORING 
PAPER U SCORE: V2 



<dusx tftu f+n &Lf^ ^,1 tsa4&& (reuuAA^tr^ ~% kL 
<L+«Ad *M*et W • ^ <LM/H ^^SX^fAc 
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PRIMARY TRAIT SCORING 
PAPER W SCORE: 4 



^^^^ £ ^dju^^uu^ /u^jC 





v ^ 
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APPENDIX D 
Examples of Writing Prompts 

Following are six sample writing items that you may wish to use with your students as is, 
or with some modification. Sources for obtaining additional items follow. 

1 • Pretend that you are a pair of tennis shoes. You've done all kinds of things with your 
owner in all kinds of weather. Now you are being picked up again by your owner. Tell 
what you, as the tennis shoes, think about what's going to happen to you. Tell how 
you feel about your owner. (NAEP) 

2. Imagine that your principal asked for suggestions about how t6 make things better in 
your school. Write a letter to your principal telling him how just ONE thing you think 
should be changed, how to bring about the change, and how the school will be im- 
proved by it. (NAEP) 

3. Pretend that one of your classmates has hurt his leg and will have to stay at home for 
a long time, ifour class would like to cheer him up. You have decided to write him a 
letter. Write the kind of letter you think he would be happy to get. (NAEP) 

4. Pretend that you are interested in a kite that is advertised in a magazine. You want 
to get one right away. All you need to do is send fVit. Write a short letter to the 
proper place (address and name of kite is provided in prompt) and order a kite 
(Adapted from NAEP) 

• 

5. One of the things you do in school is to write reports for science, social studies, and 
other subjects. Imagine that you are going to write a report about the moon for your 
science class. In the box below are some facts about the moon which you 0 can use in 
your report. You may also add other facts that you remember about the moon from 
your reading and classwork, from television, or from listening to people. 

Write your report as you would tell it to your class. Be sure to report the facts in an 
order that will be clear and that will ^aW e sense to your classmates. 

FACTS AbuuT THE MOON 

Made of rock 
Mountainous, contains craters 
Covered with dust 
No air of water 
No plant or animal life (NAEP) 

6 Describe an object (not a person, animal or event) you are especially attached to and 
tell why you feel strongly about. You might want to consider the way you discovered 
it, toe way it came into your life, or the way it has taken on meaning through time. 
(California Assessment) 
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Other potential resources for acquiring writing prompts: 



California State Department of Education. An assessment of the writing performance of 
California high school seniors. Sacramento, CA: Office of Program Evaluation and 
Research, 1977. 

Frederick, Vicki. Writing assessment research report: A national survey. Madison, WI: Wis- 
consin Department of Public Instruction, 1979. • 

National Assessment of Educational Progress. The second national assessment of writing: New 
and reassessed exercises with technical information and data. Denver, CO: Education 
Commission of the States, 1978. 4 

Office of the Los Angeles County Superintendent of schools. A common ground for assessing 
competence in written expression. Los Angeles, CA: Division of Curriculum and Instruc- 
- tional Services, 1979. 

Wisconsin Department of Public Instruction. Wisconsin pupil assessment program: Writing 
tests. Madison, WI: Wisconsin Department of Public Instruction. 
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APPENDIX E 
Additional Readings 

This bibliography includes a number of references which may be of additional assistance 
in understanding procedures for developing and using direct assessment to evaluate writing 
proficiency. 

Breland, Hunter M. and Judith L. Gaynor. A comparison of direct and indirect assessments of 
writing skills. Journal of Educational Measurement, Summer 1979, 16 (2), 119-128. 

Results of an investigation comparing direct and indirect assessments of writing ability 
of 2000 freshman students are reported and discussed, with emphasis on the validity 
and reliability of each testing approach. For purposes of this study, the direct assess- 
ment involved 20-minute essays on three different topics. Each essay was rated in- 
dependently by two different experienced readers using the holistic scoring method. 
The College Board's Test of Standard Written English was administered as the indirect 
measure. Results from three types of analysis ^how a substantial correlation between 
direct and indirect assessments of writing skill. The report also includes a review of past 
research and writing concerning direct and indirect assessment of writing skill, with 
arguments supporting each approach. In addition, it cites ways in which procedures for 
direct assessment have improved over recent years. 

Brown, Rexford. What we hum now and how we could know more about writing ability in 
America. Paper presented at the NIE Conference on Writing, Los Angeles, CA, June 
1977. 17 pp. 

The author discusses current state of writing quality in America, and suggests means 
for expanding the knowledge base. Intuitive and empirical bases precipitating students', 
teachers', and parents' awareness of student writing incompetence are explored, with 
emphasis on the deficiencies of objective writing tests as means for disclosing evidence 
of poor writing. Findings from the National Assessment of Educational Progress 1969 c 
and 1973-74 writing assessments are discussed and compared, revealing a decline in the 
quality of student writing from the first to the second asessments. Methods for scoring 
£nd evaluating the student essays are described. In addition, data from the two 
assessments and information from objective tests are summarized, suggesting that 
there is little accurate, unbiased, concrete information about writing achievement in 
America. The author concludes with suggestions for the development of "ideal" in- 
struments and procedures for the direct assessment of writing proficiency. 

Cooper, Charles R. and Lee Odell. Evaluating writing: Describing, measuring, judging. Ur- 
bana, IL: National Council of Teachers of English, 1977. 156 pp. 

This book includes a collection of chapters by nationally known writing specialists on 
the assessment of writing proficiency. The publication resulted from a 1975 NCTE con- 
ference entitled w Describmg Writing and Measuring Growth in Writing." Included are 
chapters on: (1) holistic sroring in which procedures, uses, and limitations are described 
with several examples; (2) primary trait scoring, also describing uses and procedures 
with examples; (3) use of computers to equate maturity of word choice in evaluating 
essays; (4) the role of syntactic structures ii^ evaluating writing skill; (5) procedures for 
measuring changes in intellectual processes as part of writing growth assessment; and 
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(6) the role of peer- and self-evaluation in the measurement of writing skill. All chapters 
are very practical in their orientation and provide comprehensive lists of references for' 
the interested reader. 

Diederich, Paul B. Measuring growth in English. Urbana, IL: National Council of Teachers of 
English, 1974. 103 pp. 

This publication describes the research that led to the development of current ap- 
proaches to writing assessment. The process advocated calls for the independent 
reading and rating of essays by at least two different teachers. The author describes 
how this rating can increase the reliability of assigning appropriate grades to students 
and improve the measurement of student growth in English. In addition, Diederich 
discusses the identification of factors, such as ideas, organization, and mechanics that 
contribute to judgments of writing ability, the effects of rater bias on grading paoers, 
methods for measuring improvement in writing, the use of personal versus staff 
grading, methods of computing the reliability of essay and objective tests and a plan for 
initiating staff grading of test essays. A series of highly practical appendices also ac- 
companies the book. They address the following issues: identifying and describing 
papers rated high, middle and low on eight writing-qualities; outlining topics for test 
essays; suggesting objective test items; and listing % items that the author considers 
important for students to learn about writing. This book is highly recommended for 
those interested in the measurement principles upon which writing assessment is based. 

Fowles, M.E. Manual for scoring the uniting sample. Analytical scoring, holistic scoring. 
Princeton, NJ: Educational Testing Service, 1978. 19 pp. 

This manual covers analytical and holistic scoring procedures used in evaluating the 
writing exercise component of the Basic Skills Assessment Writing Program. Brief 
descriptions of the development and composition of the multiple choice and writing 
sample parts of the program are included; however, the major portion of the manual is a 
comprehensive guide to procedures used in a large-scale assessment of student writing. 
-Advantages and disadvantages of both scoring, methods are discussed. Instructions to 
the testing coordinator cover steps in selecting and training chief and assistant chief 
readers, scorers, and aides, as well as recommendations for scheduling scoring ses- 
sions, making room arrangements,' duplicating materials, preparing test books for scor- 
ing, and arranging for security of exercise booklets. Sample writing exercises are in- 
cluded. Sections with instructions to readers, scorers, and aides can be separated, 
photocopied, and given to appropriate participants involved in scoring sessions. 

Klaus, C.H., et al. Composing childhood experience: An approach to writing and learning in the 
elementary grades. St. Louis, MO: CEMREL, Inc., 1979. 108 pp. 

This guiuebook describes an approach designed to help teachers in teaching expressive 
writing to elementary students. Much attention is focused on developing children's ex- 
pressive writing skills th rough specific writing activities and exercises. In addition, the 
report presents a carefully developed system for evaluating student responses to these 
tasks using the primary trait scoring method, suggests how the evaluation system can 
be used to help students improve their writing, and shows how to develop a sequence of 
expressive writing tasks, incorporating continuous evaluation within overall instruc- 
tional plans. Results of the 1969-70 and 1973-74 National Assessment of Educational 
Progress (NAEP) Writing Assessment are presented along with examples of writing 
exercises, a complete description of the Primary Trait System of scoring, and instruc- 
tional comments for each of the sample items. Means for comparing classroom writing 
performance to NAEP results are provided. 
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National Assessment of Educational Progress. Write/rewrite: An assessment of revision skills. 
Denver, CO: NAEP, 1977. 29 pp. 

Selected results from the second National Assessment of Writing are reported and 
discussed. Data are based on detailed analyses of two writing exercises. The first exer- 
cise required 9- and 13-year-olds to write and revise a school report, and the second re- 
quired 17-year-olds to write and revise a letter in response to a practical, reasonably 
complex problem. Aproximately 2500 individuals at each age level responded to one of 
the exercises. Samples of both writing exercises are provided, along with a description 
of the multiphase essay scoring system used by NAEP. An analysis of students' revision 
skills reveals that at all three ages, students made more stylistic, informational and 
mechanical changes than any other type of revision; and that overall, students' revi- 
sions had little effect on the quality of final drafts. Information about specific types of 
revisions made by students is broken down by geographic region, age, sex, race, paren- 
tal education, and community type. Detailed comparisons between original and final 
drafts are also included. 

Spandel, Vicki and Richard J. Stiggins. Direct measures of writing skill: Issues and applica- 
tions. Portland, OR: Northwest Regional Educational Laboratory, 1980. 64 pp. 

This document provides a comprehensive review of all statewide writing assessment 
currently underway, as well as specific guidelines for developing, administering, and 
scoring a writing assessment based on actual samples of student writing. The relative 
merits of direct measures of writing skill (writing samples) versus indirect measures 
(objective tests) are explored briefly. Technical considerations - reliability and validity 
issues - in ensuring quality assessment are addressed. Procedures for developing 
wnting exercises or prompts are reviewed. And detailed descriptions are provided for 
three popular essay scoring strategies: holistic, analytical, and primary trait scoring. 
The three strategies are compared in terms of development, utility and cost. The b -k 
concludes with a chapter on adapting writing assessment to specific testing purposes. 
Eight specific educational decisions that can be facilitated via writing tests are describ- 
ed, and specific guidelines are offered for designing and conducting an assessment 
tailored to fit each decision. 

Mullis, Ina. The primary trait system for scoring uniting tasks. Denver, CO: National Assess- 
ment of Educational Progress, 1975. 32 pp. 

The author presents the rationale and procedures involved in implementing the Na- 
tional Assessment of Educational Progress (NAEP) primary trait system for scoring 
writing samples. As defined here, primary trait scoring involves analyzing a piece of 
wnting to determine whether it manifests the traits necessary to have the desired effect 
upon the intended audience. Sample materials and responses from the 1974 national 
assessment of writing are included. The author explains that primary trait scoring 
evolved in response to desire for a scoring method that could be explained meaningfully 
in reports, defined precisely enough to enable replication of scoring, and interesting and 
useful enough to provide worthwhile information to educational decision makers. 
Development of writing tasks, trait-specific scoring guides and procedures, means of 
identifying desirable writing skills to be measured, and procedures for systematic train- 
ing of readers are all discussed. 
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APPENDIX F 
Using the Handbook in a Group Workshop 

In developing this handbook, CAPT attempted to provide materials that could be used 
not only individually, but in group teacher workshops as well. Pilot testing completed prior to 
pub .cation verified, through test scores and teachers' reactions, that the materials can be effec- 
tively used in this way. Therefore, some educators may wish to consider using the handbook as 
a basis for inservice training. In addition, teacher educators might consider using the materials 
m preservice training programs. The handbook is conveniently subdivided to fit a variety of 
class or inservice workshop agendas. 

A highly trained writing assessment expert is not needed to lead the workshop. It is 
recommended, however, that an individual go through a workshop as a participant before at- 
tempting to lead a workshop. In addition, workshop leaders should have experience teaching 
writing in the classroom. / 

The workshop leader can either prepare an oral presentation on the easy-to-understand 
writing sample scoring procedures, or have trainees go through the handbook at their own 
pace. Since the practice exercises are designed for use without supervision or assistance, the 
only essential group interaction is discussion of the classroom applications presented in 
Chapter 5 of the handbook. 

Following are three sample workshop agendas that illustrate the flexibility of the 
mate-^als. It is not necessary that you follow any of these agendas exactly; a review of the agen- 
das will, however, give you an accurate estimate of the time required for suggested workshop 
activities. 

AGENDA #1 
Four-Hour Workshop 

Introductory Comments 15 m j nu te S 

Overview of the Workshop 
Review of the Writing Assessment Process 
Other Comments by Workshop Leader 
(Handout: Chapter 1 of Handbook) 

Holistic Scoring 30 minutes 

Description of Scoring Process 
Completion of Practice Exercises 
Discussion and Questions 
(Handout: Chapter 2 of Handbook) 

Analytical Scoring 30 minutes 

Description of Scoring Process 
Completion of Practice Exercises 
Discussion and Questions 
(Handout: Chapter 3 of Handbook) 

BREAK 15 minutes 
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Primary Trait Scoring 30 minutes 

Description of Scoring Process 
Completion of Exercises 
Discussion and Questions 
Handout: Chapter 4 of Handbook) 

Classroom Applications 15 minutes 

Review of Instructional Uses (With Examples) 
Review of Testing/Evaluation Uses (With Examples) 

Small Group Discussion , 45 minutes 

Have teachers break up into groups of four or five 
to generate lists of additional uses. 

Small Group Reports and General Discussion 60 minutes 

Have reporters from each group share the list of uses generated; 
leader might create a written list of all unique ideas to be shared 
with participants after the workshop. (Handout) 

Alternative Small Group Activity 60 minutes 

As an alternative or extension of small group activities, ask 
one member of each group to demonstrate one classroom adaptation 
to the larger group. Demonstrations should be conducted as if 
other workshop participants were students. 

AGENDA #2 
Two Two-Hour Sessions* 

Present scoring procedures and practice for all three methods (no break) 

Present classroom applications and conduct small group discussions (no break) 

AGENDA #3 



Five One-Hour Sessions* 

. Introduction and Holistic Scoring 

Analytical Scoring 

Primary Trait Scoring « 

Review of Classroom Applications and small group discussion to generate new ideas 
(Participants might be asked to generate additional ideas or develop demonstrations 
between sesions.) 



2 hours 
2 hours 

1 hour 
1 hour 
1 hour 
1 hour 

1 hour 



Discussion of Classroom Applications 

Whenever ideas for classroom instructional use are identified that are not included in this 
handbook, CAPT would be interested in learning about them. Ideas generated by handbook 
users will be disseminated in future CAPT publications. 



•It is recommended for the sake of continuity that, if possible, sessions be conducted no more than 24 hours apart. 
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APPENDIX G 
Answers to Pre/Posttest 



/. List the major steps in the writing assessment process. 

A: Clarifying the reason for assessment, planning the writing exercise, administering 
the writing assessment, scoring results (p. .4 ). 



A: Primary trait (p. 5 , p. 39). 

5. Which scoring approach takes the most time to use? 
A: Analytical (p. 25). 

6. Which scoring approach takes the least time to use? 

A: Holistic (p. 5). 

7 . Which scoring approaches are based on prespecified performance criteria ? 

A: Analytical and primary trait (p. 19, p. 39). 

8. If you wanted to evaluate specifically how well students follow proper format in 
writing a business letter, which scoring approach would you use? 

A: Primary trait (Suggestion 9-10, p. 60). 

9. Which scoring approach provides the most information about specific strengths 
and weaknesses in s h 'dent writing? 



A: Analytical (p. 57). 
10. What is a "range finder" paper? 

A: A range finder is a model paper selected for its representativeness for use as a scor- 
ing guide (p. 5, p. 7). 
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11. Which of the scoring approaches is probably most subject to a "halo effect"? 
A: AnsJytisal (p. 19). * 

12. Probably the best way to use writing assessment in the classroom is to show 
students how you are going to score their papers. 



^iJ' alse; students learn more through involvement than mere demonstration (d 57 
59n). • vr« «• » 
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